Blocking AI Scrapers Is More Useful Than Playing Games with Them

There is a certain appeal in wasting the time of an AI scraper that ignored your site’s boundaries. Developers have proposed giant files, poisoned content, honeypots, and special responses. Some ideas are funny; not all of them are sensible to run on a normal site.

Send a bot toward a giant file

Robb Knight suggested pointing AI scrapers from robots.txt toward an enormous file full of useless data. It makes the crawler spend bandwidth and processing time on nothing.

It also makes your server spend bandwidth. I would not serve a 10 GB joke from a site where traffic or hosting costs matter. It is a protest, not a practical access-control mechanism. Chris Ferdinandi describes a related “poisoning the well” idea.

State the boundary clearly

Eric W. Bailey writes about consent and LLM scrapers, including metadata such as robots.txt rules. These rules express the site’s preference, but they only work when a crawler chooses to respect them.

Block known crawlers

If the goal is to reduce scraping, blocking is more useful than confusing. Ethan Marcotte documents his approach using robots.txt and server controls.

No method is complete. User-agent names can be spoofed, IP ranges change, and an unidentified crawler looks like ordinary traffic. I would start with known bots, check logs, and avoid rules that accidentally block search engines or real users.

Return a bot-specific response

Some developers return different content to identified AI bots. This GitHub commit is one example.

Bot-specific responses are easier on a server than giant downloads, but they still depend on reliable identification. They can also become another branch of application behavior that needs maintenance.

Keep the joke harmless

An error message or small Easter egg is harmless if it does not waste resources or affect accessibility. sizeof(cat) collects another approach to blocking ChatGPT scraping.

What I would actually do

• Publish clear robots.txt rules for named crawlers.
• Use host or firewall controls when a crawler ignores them.
• Review logs before and after the change.
• Keep normal pages identical for real visitors.

Honeypots and poisoned responses are interesting experiments. For most sites, boring controls are easier to explain, cheaper to operate, and less likely to cause collateral damage.