Should you block AI crawlers

Guide · SEO / AEO / GEO · 5 min read · last verified 2026-07-27

Blocking AI crawlers means instructing the automated agents that collect web content for AI systems not to fetch your pages — most commonly through directives in your robots.txt file, sometimes through server rules that refuse the requests outright. Whether you should is genuinely contested, and the honest framing is that both positions give something real away. Blocking asserts control over how your content is used, at the price of absence from answers your buyers increasingly read. Allowing trades your content for presence in those answers, on terms you do not set and cannot fully audit. No configuration gets you both, so the real question is which loss your business can better afford.

How robots directives work — and where they stop

Robots.txt is a plain-text file at the root of a domain that names crawlers and states what they may fetch. A file that blocks one AI crawler while leaving the site open to everything else looks like this:

User-agent: ExampleAI-Bot

Disallow: /

User-agent: *

Allow: /

Two caveats matter more than the syntax. First, the file is advisory. Compliance is voluntary: major operators generally state that they honor it, and observed behaviour largely supports that, but nothing enforces the directive, and reports of inconsistent compliance surface regularly. Robots.txt is a posted sign, not a locked door. The stronger alternative — refusing requests at the network level — actually withholds content, but is heavier to maintain and can be sidestepped by a crawler that changes how it identifies itself.

Second, and this is the part that ages fastest: the crawler names are moving targets. GPTBot is a widely cited example as of this writing, and several operators currently publish separate names for crawlers that collect training data and crawlers that fetch pages to ground a live answer. But the names, the operators, the training-versus-retrieval distinction, and the opt-out mechanics have all changed before and will change again. Any robots.txt you write today encodes a snapshot of the current crawler ecosystem. Treat it as configuration that needs scheduled review, not a decision you make once.

The case for blocking

The strongest argument is control. If your content is your product — proprietary research, paywalled analysis, data you license — then letting AI systems ingest it and answer with it can substitute for the visit or the license that funds the work. For a business in that position, allowing crawlers restates the problem it is trying to solve. Blocking is also a negotiating posture: some publishers block precisely to create the conditions for a licensing conversation, and keeping that option open costs little. There are quieter reasons too. Crawler traffic consumes serving resources, and some teams object on principle to uncompensated use — a position that needs no commercial justification to be legitimate.

What blocking is unlikely to do — at least today — is remove you from AI answers. Engines still encounter your company through every page you do not control: reviews, directories, forum threads, comparison posts, news coverage. These third-party pages are citation surfaces in their own right — see What is a citation surface — and they currently stay in the corpus whether or not your own site does. Blocking silences your version of the story while leaving everyone else's version intact.

The case for allowing

The argument for allowing is that AI assistants have become a place where buyers form impressions and shortlists — an observed shift, uneven across industries and hard to see in analytics, which is part of why AI referral traffic is undercounted. Absence from a surface your buyers consult is a cost even when no dashboard shows it.

There is a sharper version of this argument. If assistants are asked about you and your pages are not readable, the answer gets assembled entirely from third-party material — often older, thinner, and occasionally wrong. Correcting a wrong record is harder when your own authoritative pages are the ones you have withheld; What to do when AI cites outdated information about you walks through that repair, and it leans on having current, crawlable pages of your own. Blocking removes your best evidence from the room where the argument about you happens.

The B2B asymmetry

For a B2B software vendor, the calculus differs from a publisher's because content is rarely the product. Your docs, integration pages, and explainers exist to make buyers confident enough to talk to you; they were already free, and the return was never the pageview — it was the impression. Meanwhile your buyers ask assistants about your category and about you specifically, whether or not you participate — How enterprises discover vendors with AI describes that behaviour. For a vendor, blocking mostly withholds evidence from a conversation that proceeds without it. For a content business whose revenue is the content, the same act protects the asset. Same directive, opposite meaning.

One caution if you allow: being unblocked is not the same as being readable. Rendering and structure decide what a crawler actually gets — How JS rendering affects what AI can read covers the difference.

A narrower middle path

The choice is not binary. Where operators distinguish training crawlers from retrieval crawlers, you can allow the fetches that ground live answers while opting out of training — with the standing caveat that this distinction is only as durable as the published policies behind it, and those policies are revised without notice. You can also split by section: keep docs, product, and integration pages open because they do buyer-facing work, and block a premium research library because it is the paid asset. Partial policies need the same scheduled review as full ones — a policy built on a crawler list from last year protects nothing.

Deciding, then revisiting

A workable decision test: does your content earn revenue directly, through access or licensing, or indirectly, through buyer confidence? Direct earners have a real case for blocking and for the negotiating room it preserves. Indirect earners — most B2B vendors — usually have more to lose from absence than from ingestion. Whichever you choose, watch the consequences rather than assuming them: ask the assistants your buyers use what they say about you, note which sources they lean on, and recheck after policy changes. Teams use Magrios to track which of their pages assistants actually cite and whether answers about them drift, which turns this from a one-time bet into a monitored position. Revisit quarterly. The crawler names will change, the opt-out mechanics will change, and a decision this contested deserves to be re-decided on current facts.