Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it

2026-11-15来源 dev.toEnglish

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com

A couple of years ago, robots.txt was mostly about search engines. Now a typical site's logs include a whole list of AI-related crawlers: OpenAI's, Anthropic's, Common Crawl's, Perplexity's, Apple's, ByteDance's and others. Many sites added blanket blocks in a hurry. Others never touched their robots.txt.

Should you block them? I think this is a genuinely open question, and the answer has changed as the bots themselves have changed. Let's argue about it.

First, they're not all the same bot

The big operators now publish several user agents with different purposes. Roughly:

Training crawlers collect pages to train models: for example OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot (used by many model builders).
Search crawlers index pages so an AI product can cite and link to them in answers: for example OAI-SearchBot, PerplexityBot, Claude-SearchBot.
User-triggered fetchers load a page because a person asked the assistant to read it right now: for example ChatGPT-User, Claude-User.
Control tokens that aren't crawlers at all: Google-Extended and Applebot-Extended are names you can disallow in robots.txt to opt out of your content being used for those companies' AI models, without affecting Google Search or Apple's search features.

Names and purposes change, so check each operator's current documentation before relying on a list. The point is that "block AI" now means several separate decisions.

The case for blocking

Consent. Your content was written for readers, and nobody asked whether it could be used to train a commercial model. Blocking training crawlers is the clearest signal you can give.

No traffic in return. Training a model on your article doesn't send anyone to your site. If the model can then answer the question your article answers, it may reduce your traffic.

Server cost. Some AI crawlers are aggressive, request huge numbers of pages, and ignore crawl-delay hints. Smaller sites have seen real bandwidth and CPU costs from them.

Leverage. Publishers negotiating licensing deals have argued that blocking is the only bargaining position content owners have.

The case for allowing

AI answers are a discovery channel now. People ask assistants for recommendations, documentation and comparisons. Products that cite sources can only cite pages their search crawlers can read. Block the search agents and you're invisible there.

Developer content wants to be found. For documentation, open source projects and developer tools, being the source an assistant learned from, or cites, is often the whole point of publishing.

Blocking doesn't remove what's already collected. Pages crawled before you added the rule may already be in datasets. A new block only affects future crawling.

robots.txt is voluntary. Well-behaved crawlers follow it. Badly behaved ones don't, and some scrapers simply pretend to be a browser. A block mostly affects the operators who were already playing fair.

Before you tighten robots.txt, it helps to see what is already blocked. seotto.com checks for indexing blocks and robots regressions, including rules you didn't mean to ship.

Where I've landed

For most sites, I'd make the decision per purpose, not per company:

User-triggered fetchers: allow. A person asked an assistant to read your page. That's closer to a browser visit than to crawling, and blocking it mainly frustrates your own readers.
AI search crawlers: allow, if being cited matters to you. For most commercial and developer sites, it does. Check your logs and analytics for referrals from AI products before deciding.
Training crawlers: a values decision. There's no technical right answer. Blocking GPTBot, ClaudeBot, CCBot and setting Google-Extended and Applebot-Extended to disallow is a coherent position; so is allowing them. Decide deliberately and write down why.
Aggressive crawlers: rate limit, don't just disallow. If one bot is costing you real money, a rate limit at the CDN or server is more reliable than hoping it reads robots.txt.
Never block Googlebot or Bingbot by accident. A blanket User-agent: * disallow added in a hurry to stop AI bots also removes you from regular search.

An example of the split approach:

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: /

Search and user-triggered agents fall under * and stay allowed.

Check what your robots.txt actually says today

Before changing anything, fetch your live file and test a few agents against it. Python's standard library can do it:

from urllib.robotparser import RobotFileParser rp = RobotFileParser("https://example.com/robots.txt"); rp.read() for ua in ["Googlebot", "Bingbot", "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "CCBot"]: print(f"{ua:15s}", rp.can_fetch(ua, "https://example.com/blog/some-post"))

It's a quick way to catch the worst outcome, which is blocking Googlebot while trying to block something else.

Revisit it every few months

Whatever you decide, put a reminder in the calendar. The list of crawlers, what each one is used for, and the products that cite sources are all changing quickly. A rule written for last year's bots may block a search agent that didn't exist then, or miss a new training crawler entirely. A quarterly look at your logs and at each operator's current documentation keeps the file honest.

The questions I can't settle

Partial blocks. Should you allow AI crawlers on marketing pages and docs but block them on paid or premium content?
llms.txt and similar files. Do they help, or are they another file nobody reads?
Measuring the trade-off. Is anyone seeing meaningful referral traffic from AI search products, enough to justify allowing their crawlers?

Before deciding, look at what these bots actually do on your site. How much of your crawl traffic they make up, and which pages they hit, is a ten-minute log query. Measuring that kind of thing from real logs, rather than guessing, is what we prefer at seotto.com.