AI crawlers are already at your door. Your robots.txt can't tell them apart
By Lindsey J
AI crawlers already request pages from your site every day. Most robots.txt files give the same answer to all of them: allow everyone, or block everyone. That one answer is still a decision, even when nobody chose it on purpose. A crawler that trains a model, a crawler that answers a live question, and an agent fetching a page for one specific person are not the same visitor. Treating them the same throws away the two that can send you a buyer, in order to block the one that only trains someone else's product.
Three different bots share one robots.txt line
A training crawler, such as OpenAI's GPTBot, collects pages to improve a future model. OpenAI's own documentation states that disallowing GPTBot in robots.txt signals that a site's content should not be used to train its models.
A retrieval crawler works differently. It fetches a page at the moment a real person asks an AI assistant a question, then returns an answer that can cite your page and link back to it. A third kind of request works differently again. It is an AI agent acting on behalf of one specific person who already has your page open, checking a price or filling in a form on that person's own instruction. Each of these three deserves its own answer, not one shared Allow or Disallow line.
Why blocking AI crawlers became a real decision in 2025
Cloudflare moved its AI Crawl Control product out of beta in August 2025. The update let every paid customer block a named AI crawler. It could also send that crawler an HTTP 402 Payment Required response carrying a licensing contact instead of a blank refusal. Cloudflare reported that its customers were already sending more than one billion of those 402 responses on an average day, as of that announcement. That figure describes how many sites were already making an access decision at scale. It says nothing about how much traffic any one crawler was sending back in return. The decision had already become routine. Most sites still had not made it on purpose.
A decision rule for which AI crawlers to let in
Sort every AI bot requesting your site into one of three tiers before you write the robots.txt line. Tier one is a crawler whose documented purpose is training data collection, with no live citation back to you. Block it, or route it to a paid licensing response if the volume justifies the cost of a tool built for that. Tier two is a retrieval crawler that can cite your page and link a person back to it inside an AI assistant's own answer. Allow it. Treat it the way you already treat a search engine. Tier three is an agent carrying a real person's own session, already using your product, asking a page to load on that person's behalf. Always allow it. Blocking it blocks your own customer.
Once the access decision is made, a separate file does the next job. The llms.txt specification gives the crawlers you have allowed a short, curated markdown map of your site, instead of leaving them to parse the same rendered HTML built for a browser. A related piece on this site covers the markup a product page needs once an agent can read it. llms.txt is the layer above that: which pages an agent should bother reading at all.
Picture a B2B software company whose robots.txt blocks every bot with the word "bot" in its name, written once during a security review and never revisited. A buyer asks an AI assistant to compare three vendors on price and support hours. The assistant retrieves the two competitors' pricing pages without trouble. It cannot retrieve this company's page, because the blanket rule caught the retrieval crawler along with the training one. The assistant drops the company from its answer rather than guessing at a price it cannot see. The fix is checking each blocked user agent against its documented purpose, then allowing the ones that cite and link, before the next deal is lost to a rule nobody meant to write that broadly.
Self-assessment: does your site have a crawler policy or a default
- Do you know which named AI crawlers, such as GPTBot, are currently allowed or blocked on your site, or is the answer whatever the template shipped with
- Does your robots.txt tell a training-only crawler apart from a retrieval crawler that cites and links back to you
- If a buyer asked an AI assistant to compare you with a competitor today, could the assistant actually read your pricing or product pages
- Do you have a markdown-readable map of your key pages ready for the crawlers you already allow
FAQ
Does blocking AI crawlers stop an AI assistant from mentioning my brand at all
Blocking a training crawler via robots.txt signals that your pages should not be used to train that one bot's models. A live retrieval feature inside an AI assistant often runs a separate bot with its own user agent. Check the specific crawler's name before assuming one Disallow line covers every AI product that could mention you.
Is llms.txt a replacement for robots.txt
No. Robots.txt decides who gets in. llms.txt only helps the agents you have already allowed find and read your site efficiently, per the specification.
Who should own this decision, marketing or engineering
Marketing decides which bots should be allowed to represent the brand inside an AI assistant's answer. Engineering implements that decision in robots.txt and keeps the crawler list current as new AI agents launch. Neither owns it alone, because the list of bots changes faster than either team's usual review cycle.
If you do not know which AI crawlers can currently read your product pages, that is worth finding out before a buyer's AI assistant finds out for you. Talk to us about an agent-readiness review.
