crontent

Your robots.txt can kill Claude visibility faster than bad content can

Crontent

A public website isn't the same as a visible website inside Claude. Anthropic now says one blocked bot can stop Claude from fetching your page for a user, while another can stop it from indexing your site for search in the first place.

How do I show up in Claude search and answers?

Allow the Anthropic bot that matches the job you want done. In Anthropic's Privacy Center, the company splits its bots into three roles: ClaudeBot for model training, Claude-User for user-requested retrieval, and Claude-SearchBot for search indexing. Those are separate switches, not one blanket "Claude can access my site" setting.

That matters because the failure mode is quiet. You can block Claude-User and still have a perfectly public site in Google. But Anthropic says disabling Claude-User "prevents our system from retrieving your content in response to a user query," which can cut your visibility when someone asks Claude about your product or pastes in your URL. You can also block Claude-SearchBot and lose indexing for Claude's search layer, even while Google and Bing keep crawling you like normal.

If you run a tiny SaaS, treat those user agents like production dependencies. Same bucket as your sitemap, canonical tags, SSL, and uptime checks. If the bot that needs to fetch you can't get in, your content won't matter much because Claude can't see it when the user asks.

Anthropic split one AI crawler into three real dependencies

Anthropic made the separation explicit on 7 April 2026 in its crawler doc at the Anthropic Privacy Center. ClaudeBot is about future model training data. Claude-User is about live requests from users. Claude-SearchBot is about improving search results.

That means you can make a deliberate tradeoff instead of a vague one. If you don't want training access, block ClaudeBot. Fine. But if you also block Claude-User and Claude-SearchBot because some firewall template lumped every AI bot together, you've also told Claude not to fetch your pages for users and not to index them for search quality.

For a small team, the practical checklist is boring:

  • check robots.txt for all three Anthropic user agents
  • check your CDN, WAF, or bot protection rules
  • check server logs to see whether requests get through
  • check whether a hosting security preset is denying them by default

"Public on the web" is not the same as "reachable by the bot that answers the question." That's the part people miss.

Blocking Claude-User breaks the exact moment a buyer asks about you

Claude-User is the bot that handles user-directed retrieval, according to the Anthropic Privacy Center. Anthropic says disabling it "prevents our system from retrieving your content in response to a user query." That's the scary one for founders because it hits the highest-intent moment.

Say a buyer asks Claude, "compare Product A and Product B," or pastes your docs URL and asks for a summary. If Claude-User can't fetch the page, you don't just lose traffic. You lose the chance to have your own page inform the answer at the moment someone is evaluating you.

Claude-SearchBot is a different problem. Anthropic says disabling that bot "prevents our system from indexing your content for search optimization," which may reduce visibility and accuracy in search results. So there are really two ways to disappear:

  1. Claude can't fetch your page when a user asks
  2. Claude hasn't indexed enough about your site to surface it well in search

Neither problem looks obvious from your normal SEO dashboard. Google Search Console won't warn you that Claude-User got blocked at the firewall.

Pasted rewrites are a weak content strategy inside Claude anyway

Anthropic's own help docs say Claude may block outputs that look like copyright regurgitation. In the Anthropic Help Center, the company says these errors "generally arise from Anthropic's efforts to prevent Claude from being used to replicate or regurgitate pre-existing materials."

That's bigger than a prompt bug. If your content pipeline depends on pasting source text into Claude and saying "rewrite this," you're building on a behavior Anthropic is openly trying to limit. Sometimes you'll get a refusal. Other times you'll get flat, over-sanitized copy because the model is trying not to drift too close to the source.

The better play for a tiny SaaS is simple:

  • feed Claude your notes, screenshots, product changes, support themes, and test results
  • ask it to structure and sharpen your thinking, not mimic someone else's page
  • publish pages you want Claude-User and Claude-SearchBot to access cleanly

If you want Claude to surface your site, do the boring plumbing first. Then give it original material worth surfacing. Check robots.txt, logs, and firewall rules this week. One blocked bot can erase you faster than one mediocre blog post can.

Sources