Designing an AI-crawler access policy: citation versus training
Guide · SEO / AEO / GEO · 4 min read · last verified 2026-07-23
An AI-crawler access policy decides, bot by bot, whether your public pages may be fetched to answer a user's question right now, collected to train future models, or both. The trade is easy to state and impossible to settle generically: blocking training crawlers withholds your content from model building; blocking retrieval crawlers removes you from the answers your buyers are reading today. Magrios allows both classes deliberately — our robots.txt names nine AI agents and grants each one access — because for a company whose measured problem is that AI systems deny it exists, citation is the distribution channel.
What do the major AI crawlers actually do?
The table below states each agent's purpose as its operator documents it. That framing is deliberate: what a bot does is the operator's claim, and compliance with robots.txt is also the operator's claim — robots.txt is a published request, not an enforcement mechanism.
| Agent | Operator | Documented purpose | What blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Crawling for model training | Presence in future model knowledge |
| ChatGPT-User | OpenAI | Fetches a page when a user's request needs it | Being readable in live ChatGPT sessions |
| OAI-SearchBot | OpenAI | Crawling for ChatGPT search results | Appearing and being cited in ChatGPT search |
| ClaudeBot | Anthropic | Crawling web data for its products | Presence in Claude's knowledge and retrieval |
| anthropic-ai | Anthropic | Legacy training-control token | Redundant control; kept for completeness |
| PerplexityBot | Perplexity | Indexing for its answer engine | Citations in Perplexity answers |
| Google-Extended | Google | Control token governing use of crawled content in its AI models — not a separate crawler, and per Google it does not affect Search ranking | AI-model use of your content, while classic Search continues |
| Bingbot | Microsoft | Bing's search index, which also feeds Copilot | Bing rankings and Copilot answer presence |
| CCBot | Common Crawl | Building an open crawl corpus, widely used in model training | Presence in datasets many labs train on |
Citation versus training: which axis are you actually deciding?
Retrieval-time access drives citations now: an answer engine fetches your page, quotes it, and links it while the buyer watches. Training-time access shapes what models simply know later: assistants answer many questions with no browsing at all, from model knowledge. Blocking training crawlers therefore does not only affect some abstract future model — it plausibly suppresses your presence in the from-memory answers buyers get. We label that a hypothesis: the mechanism is not publicly documented, but it is grounded in the observable fact that chat models name companies without fetching anything.
The two axes also compound. A brand absent from training data depends entirely on retrieval to appear; a brand absent from retrieval depends entirely on what the model already believes. How AI assistants choose their sources covers the retrieval half; brand mentions vs citations covers why the two outcomes are worth tracking separately.
What does a worked policy look like?
Ours, in production. The policy itself is one rule set: public pages — the marketing site, pricing, FAQ, trust page, the knowledge library — are allowed for every agent. Customer report paths, share links, API routes, and the scan surface are disallowed. A curated set of sample reports is explicitly re-allowed by exact path, using longest-match precedence, so those specific runs are crawlable while every customer report stays out.
The file then names nine AI user agents, and group-matching semantics dictate how. Under RFC 9309 a crawler obeys only the most specific User-agent group that matches it: a bot that matches a named group ignores the `*` group entirely and inherits none of its rules. So each named group restates the full rule set — the welcome and the boundaries travel together, nine times over. We state that as a lesson learned, not a flourish: an earlier version of our file gave each named bot a bare allow, which read as friendly redundancy and in fact dropped every Disallow for exactly the bots we were naming. Adversarial review caught it; the restated groups are the fix. The named blocks remain deliberate self-documentation — a future maintainer who adds a sweeping Disallow cannot silently cut off citation, because the file states the policy bot by bot — but the mechanism comes first: a named group is a complete policy, not an annotation. A robots.txt is read by more future employees than any strategy memo.
When should you decide differently?
Our choice follows from our position, not from principle. Honest decision criteria:
- Content is the product — publishers monetizing archives may rationally block training crawlers while keeping retrieval agents, since training use substitutes for their product.
- Obscurity is the problem — vendors fighting for existence in answers, as we are, should generally allow both classes; vendor sites rarely win citations even when fully crawlable, so adding access barriers only deepens absence.
- Private surfaces are non-negotiable — customer data, reports, and authenticated paths stay disallowed for every agent under every strategy. An access policy is about your public pages only.
Whatever you choose, write the policy so it reads as a policy: named bots, stated intent, private paths excluded. The file is public — treat it as documentation someone will one day quote back to you.