Most sites that block AI crawlers are blocking the wrong ones. GPTBot, ClaudeBot and Google-Extended feed model training, not the answer a buyer reads today. A separate bot, published under a different name, fetches the page a live ChatGPT or Perplexity answer actually cites. Disallow the first group and expect fewer citations, and nothing changes, because the bot that decides citations was never touched.
Buyer asks: “does ledgerline's pricing page offer a discount for annual billing”
Ledgerline's pricing page confirms a 15% discount for annual billing on every paid plan, applied automatically at checkout with no separate code needed.
This citation only happened because OAI-SearchBot could still reach the pricing page. Ledgerline disallows GPTBot site-wide to keep its content out of model training, and never checked whether that rule touched anything else.
What is an AI crawler, and why does it need its own rule?
An AI crawler is any automated bot an AI company runs to fetch web pages, and most companies that ship a chat product run more than one. One token trains a future model on whatever it reads, sometimes months before that model ships. A different token, published under an unrelated name, fetches a page only at the moment a live question needs an answer and a source to cite.
Robots.txt has no concept of "AI" built in. It only understands User-agent strings, so it cannot tell a training crawler from a citation crawler unless the file names both, correctly, under separate rules. A single Disallow: / under User-agent: GPTBot does exactly one thing: it stops GPTBot. It says nothing at all about the six or seven other tokens the same handful of companies also publish.
Robots.txt predates any of this by three decades, and it still works the same way it always did: the newest AI-specific crawlers just added themselves to the same file everyone already had. Our llms.txt piece covers the newer, AI-specific file that sits alongside it, and why robots.txt is the one that can actually make a brand invisible if it is wrong.
Which bots train models, and which decide who gets cited?
Four of the five companies below publish a training bot and a separate retrieval bot, and the two names rarely resemble each other. OpenAI runs GPTBot for training and OAI-SearchBot for the citation ChatGPT shows in a live answer. Anthropic and Apple split the same way. Google does not, and the exception is the one that catches the most people out.
| Company | Trains future models | Fetches a page to cite it right now |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Perplexity | Not separately published | PerplexityBot |
| Google-Extended | Googlebot, the same crawler that feeds ordinary search | |
| Apple | Applebot-Extended | Applebot |
| Common Crawl | CCBot, an open dataset several labs train on | Not applicable, it never answers a live question |
Google is the one company on this list that does not run a dedicated crawler for its AI surfaces. Google-Extended is a training opt-out flag, nothing more: set it, and a page stops feeding future Gemini and AI Overview training. The page an AI Overview or AI Mode answer actually cites today comes from whatever Googlebot already indexed for regular search, because Google reuses one index rather than crawling twice. Block Google-Extended and AI Overviews continue citing the page exactly as before. Block Googlebot instead, and the page disappears from Google entirely, AI surfaces and the ten blue links together, which is a much bigger decision than most teams mean to make with one line in a config file.
Does blocking GPTBot also block ChatGPT's citations?
No. Disallowing GPTBot stops a page from training a future OpenAI model, and does nothing else. ChatGPT's live citations come from OAI-SearchBot and ChatGPT-User, two separately published tokens that a GPTBot rule never touches. A brand can opt out of training entirely and still be cited in tomorrow's answer, which is usually the outcome a marketing team actually wants once someone explains the difference.
The reverse mistake happens too, just less often. A brand that genuinely wants no presence in any AI surface, training or citation, has to name every token in the table above, one User-agent block at a time. Missing even one, commonly Claude-SearchBot or ChatGPT-User because they are the least publicised of the group, leaves that one door open while every other bot is shut.
- 1
Open the live file
Visit yourdomain.com/robots.txt in a browser. A 404 means nothing is currently blocked, and every bot in the table above is free to fetch your pages.
- 2
Search for each token by name
Look for a Disallow rule under User-agent: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot or Applebot-Extended. A rule under one name never applies to another, even from the same company.
- 3
Check the edge, not just the file
A CDN or WAF rule can block a user agent before it ever reaches robots.txt. If a platform ships a single "block AI bots" switch, confirm exactly which tokens it covers before assuming the file alone tells the whole story.
Common mistakes that block citations by accident
The failures are rarely deliberate. A legal team asks engineering to "opt out of AI" and gets one line added under the wrong User-agent block, or a WAF rule written to catch anything with "bot" in the string sweeps up a citation crawler along with the scrapers it was meant to stop. Each version looks identical to a healthy robots.txt file until a page that used to get cited quietly stops.
- Copying a training-bot
Disallowrule underUser-agent: *by mistake, which blocks every crawler on the list at once instead of the one intended - A CDN's "block AI bots" toggle covering retrieval tokens nobody meant to touch
- Blocking Googlebot entirely to "stop Google's AI," which removes the brand from ordinary search results too
- Trusting a redesign checklist or a stale audit instead of checking the live file after a CDN migration or platform switch
- Assuming one
Disallowline for one bot from a company blocks every bot that same company runs
See where your brand shows up in AI search
Enter your website and get your AI Visibility Score across 7 engines. First score in under 5 minutes, no setup.
None of this shows up inside a GEO audit, which measures the answer layer: which prompts surface a brand, how it gets described, which sources an engine leans on to decide. Crawler access sits one layer underneath that, and an audit built to read finished answers has nothing to say about a page it was never allowed to reach in the first place. It belongs earlier than an llms.txt file too: a curated entry point is wasted on a crawler that never got past robots.txt to find it.
Check the live file today rather than at the next redesign. It takes about four minutes and touches nothing else on the site.
Frequently asked questions
An automated bot an AI company runs to fetch web pages, either to train a future model or to answer a live question and cite a source for it. Most companies that ship a chat product publish at least one bot for each job, under two different names.
