Stop Blocking AI Bots as One Group

A lot of small SaaS teams are blocking every bot with “AI” in the name and calling it done. That can shut off the exact crawlers that help your pages get discovered in AI answers.

How should I block AI crawlers without losing AI discovery?

Split the decision by job, not by whether a bot sounds like AI. S. Bowles’s crawler policy guide makes the cleanest version of the rule: allow search and answer discovery bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot if you want visibility, and block training bots like GPTBot and ClaudeBot if your concern is model development.

That matters because robots.txt is more precise than most teams use it. You do not need one global “AI bots yes/no” policy. You can set different rules for different named agents.

S. Bowles’s matrix breaks it into three buckets:

• Discovery crawlers: bots used for search and answer discovery
• Training crawlers: bots used for model development
• User-triggered fetchers: bots that fetch a page because a user opened a URL inside a tool

That last bucket is easy to miss. A user-opened fetch is not the same thing as broad site crawling. If you treat them as one thing, you end up making a policy decision with the wrong mental model.

Blanket blocks can cut off citations even when training is the real concern

OpenAI’s bots are a good example because the names are easy to confuse. ICODA’s guide draws a sharp line between GPTBot and OAI-SearchBot: one is tied to training, the other to search and discovery. Their point is simple. If you block both, you may stop training access, but you may also lose the crawler that helps your pages show up in ChatGPT citations.

That does not mean an allow rule guarantees citations. S. Bowles’s guide is careful on that point: robots.txt only expresses crawler access preferences. It does not guarantee indexing, rankings, citations, model inclusion, or retrieval by a user-requested tool.

But the reverse is still true in practice. If you block a discovery crawler, you are removing one path to being found. For a tiny SaaS team, that is a bad trade if your real goal was only to keep training bots out.

The better default is boring and specific:

• Allow discovery bots
• Block training bots if you want to opt out
• Decide separately on user-triggered fetchers
• Keep private content behind auth, not robots.txt

That last part also comes from S. Bowles’s guide: private files should be protected with authentication or edge authorization, because robots.txt is not access control.

robots.txt is only a stated policy, not proof of what happened

A robots.txt file tells compliant crawlers what you want. It does not tell you what actually hit your site. Search Engine Journal frames the real operator question well: should you block these crawlers, or measure whether they lead to citations, referral traffic, or conversions before deciding?

That is the right question because policy without logs is guesswork. S. Bowles’s guide says to verify observed traffic with server logs and provider-published IP ranges. The release checklist even includes an access-log check after production.

For a small team, the practical version is simple:

1. Update robots.txt with separate rules by crawler name
2. Tag and review referral traffic from AI products in analytics
3. Check server logs for the named bots you allowed or blocked
4. Note which pages they request and how often
5. Review once a month, not once a year

You do not need perfect attribution to make a better call. You just need to stop flying blind.

User-agent strings are easy to fake, so verify bot claims with IP checks

A request that says “GPTBot” is not automatically GPTBot. ICODA’s guide calls out verification with IP ranges and crawl patterns, and S. Bowles’s guide says the same thing more directly: verify observed traffic with provider-published IP ranges.

That matters because a lot of teams are making policy decisions from user-agent strings alone. If the string can be spoofed, your measurement can be wrong in both directions. You can overestimate bot activity. You can also think a blocked bot is still active when you are really seeing impersonators.

If you run a 1-5 person SaaS, the setup does not need to get fancy. Keep one short policy doc. List which bots you allow, which bots you block, and why. Then check logs against published IP data before you declare any bot “valuable” or “harmful.”

The useful move is not “block AI” or “allow AI.” It’s separating discovery, training, and fetchers, then checking your logs before you decide what stays.