How to Block AI Crawlers Without Blocking Search

If you want to stop AI companies crawling your site, block the crawler you mean—not every verified bot. Search crawlers, user-triggered fetchers, and model-training crawlers can use different identities and serve different purposes.

No method here can prevent someone from copying a public page. It controls compliant crawlers or requests you can identify.

Start with robots.txt

For a crawler that honours the Robots Exclusion Protocol, add its documented user agent to /robots.txt:

User-agent: {BOT-NAME}
Disallow: /

A 2024 list might have looked like this:

User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /

Do not copy that list blindly. Providers can add agents or separate training from search and user-requested visits. Check each provider’s current documentation, decide which access you want to refuse, and test the published file.

robots.txt is a request, not enforcement. A malicious crawler can ignore it.

Enforce the block at Cloudflare

Cloudflare introduced a verified-bot category for AI crawlers. In the WAF rule builder, the expression was:

(cf.verified_bot_category eq "AI Crawler")

Choose the rule action carefully and review the matched traffic after deployment. Product names, plan availability, and dashboard paths change, so use Cloudflare’s current bot-management documentation for the setup.

This is stronger than robots.txt because matching requests can be rejected. It also depends on Cloudflare correctly identifying the crawler; a fake or unknown agent may require rate limits or a different rule.

Be careful with IP blocks

The original version of this article linked to an OpenAI crawler and fetcher list and blocked one published network:

sudo ufw deny proto tcp from 23.98.142.176/28 to any port 80
sudo ufw deny proto tcp from 23.98.142.176/28 to any port 443

That range is historical, not a current recommendation. Provider networks can change, and a stale block may miss the crawler or reject unrelated traffic. Only use IP rules from current official ranges, automate updates if possible, and inspect their effect.

noai is only a preference

Some publishers add these non-standard tokens:

<meta name="robots" content="noai, noimageai">

There is no general requirement for crawlers to honour them. A WordPress plugin can add the tag, but installing a plugin does not turn the tokens into access control.

I would publish precise robots.txt rules first, enforce them at the edge when the crawler creates cost or ignores the request, and keep private content behind authentication. A polite meta tag can state your preference. It cannot guard the door.

← Back to all articles