I Found My Own Website Blocking ChatGPT. Then I Checked 640 Others.

2026-11-19来源 HackerNoonEnglish

In August I ran my agency's AI search checklist against our own site and got a result I didn't believe. Every request that named an AI crawler came back 403. GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, and PerplexityBot were all refused, including on llms.txt, a file that exists only for those bots to read. Googlebot got a 200, and so did plain curl.

Our robots.txt allowed every one of those crawlers by name. That didn't help, because the block sat in front of it. Cloudflare has a "Block AI bots" setting under Security > Bots that matches on user-agent before a request ever reaches the server, and it never reads robots.txt. I don't know whether it was on by default or got switched on while we hardened the zone after a DDoS attack in July. Nobody on the team remembers choosing it.

Setting it to Allow took two minutes. Two weeks later I pulled Cloudflare's per-bot logs and found PerplexityBot still failing on 24 of its 56 requests to real pages. This time the refusal came from our host. Hostinger runs an IP-reputation firewall at the origin that flags some of Perplexity's AWS addresses, and it only fires when Cloudflare's cached copy has expired. The same bot got a 200 on a cache hit and a 403 a minute later. The fix was adding Perplexity's 18 published IP addresses to the host's allow list, one at a time, because the field doesn't accept ranges.

That made two blocking layers from two vendors, and neither showed up in any file I'd normally check. I wanted to know how common this is.

What 640 sites returned

On September 6 and 7 we scanned 640 websites: 100 each in B2B software, fintech, AI companies, crypto, healthcare, and online stores, plus 40 smaller software companies for comparison. I picked every list by hand, so the numbers describe these companies rather than whole industries. Nine sites rate-limited the scanner, which left 591 we could measure.

100 of the 591 turned away a request naming GPTBot or ClaudeBot.

Industry

Sites that refused an AI crawler

Online stores

27

Healthcare

21

Crypto

17

Fintech

14

B2B software

11

AI companies

10

Only 4 of the 100 documented the decision in robots.txt. Two of those four were medical publishers, and a medical publisher keeping its articles out of AI systems is a coherent position. The other 96 fell into two groups:

  • Sites whose security challenges every automated client, AI crawler or not.
  • Sites that serve a browser normally and return 403 to an AI crawler, with nothing written down anywhere.

Two of the silent blockers in the fintech set refused one AI crawler and allowed the other. I can't think of a policy anyone would choose that produces that result.

Why a scanner's 403 can mislead you

Before calling any site a blocker, we re-tested it by hand from a second network, and the retests changed the picture. Bot-management products fingerprint a client from its TLS handshake and header order, not only its IP address and user-agent. Three fintech sites returned 403 or 429 to us from a datacenter address and from a home connection, with a Chrome user-agent and with a GPTBot one, while a real browser loaded the page fine. Those sites reject every client that isn't a full browser, AI crawler or otherwise. They may still let verified crawlers through by IP range, which nobody outside the company can test.

The reverse also happens. A request claiming to be GPTBot from an address OpenAI doesn't own can be blocked as a spoof while the real GPTBot gets through. An outside test tells you where to look, and your logs tell you what actually happened.

Test your own site in five minutes

Start from outside. This compares what a browser, three AI crawlers, and Googlebot get from the same URL:

URL=https://yourdomain.com/for ua in \ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" \ "GPTBot" "ClaudeBot" "PerplexityBot" "Googlebot"; do code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$URL") echo "$code ${ua:0:40}"done 

How to read the results:

  • Everything returns 200. Nothing refuses on user-agent. Check the logs anyway, because an IP-based block won't show up here.
  • Browser and Googlebot get 200, AI crawlers get 403. A user-agent rule refuses AI crawlers specifically. That was our Cloudflare setting.
  • Everything returns 403, the browser string included. Your bot management is fingerprinting curl, and the test says nothing about AI crawlers.
  • GPTBot and ClaudeBot get different codes. Somebody wrote a rule for one crawler and not the other.

Then find out which layer answered:

curl -sI -A "GPTBot" https://yourdomain.com/ | grep -iE '^(HTTP|server|cf-cache-status)' 

A 403 with server: cloudflare and a short "request blocked" body came from the edge. A 403 carrying your origin's server header came from behind the CDN. If the 403 only appears when the cache status reads MISS or EXPIRED, suspect the host's firewall, which is what ours turned out to be.

Then read the logs

Real crawlers don't come from your laptop, and IP-based blocks never show up in a curl test. Filter your CDN or server logs for these user-agent tokens and look at the status codes you returned over the last week: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot.

On Cloudflare, Security Events lists what the edge blocked. Refusals from the origin won't appear there, so compare the edge status with the origin status for the same requests. That comparison is how we found the Perplexity problem.

If you block AI crawlers, do it on purpose

There are good reasons to keep AI crawlers out, and a publisher selling access to its archive has one. If that's you, name the bots in robots.txt so the decision is visible, then enforce it at the edge. Know which bots you're naming. GPTBot and ClaudeBot collect training data. ChatGPT-User and Claude-User fetch a page when a person asks a question about it, so blocking those keeps your pages out of the answers themselves.

The 96 undocumented refusals are the case to avoid. Nobody decided them, they live in vendor settings nobody opens, and meanwhile someone on the marketing team is writing pages for engines that can't fetch them. Cloudflare is replacing its single AI bot toggle with separate policies starting September 15, 2026, so if your site runs on Cloudflare, open that settings page and confirm what each policy is set to.

原文:https://www.a8q9.cn/en/article/i-found-my-own-website-blocking-chatgpt-then-i-checked-640-others-e3eaff.html

相关文章