I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool
Can ChatGPT, Claude or Perplexity read your website? The first place to look is a small text file: robots.txt . I wanted to see what big websites actually say i…




