Which bots should you let crawl your website?

A bot is just software making automated requests. The same basic mechanism can index a site for search, check whether it is online, scrape every product price, or hammer a login form.

“Good bot” and “bad bot” are useful shorthand, but behaviour matters more than the label.

Bots that provide a service

Search crawlers discover pages and build an index. Monitoring bots check availability. Feed readers fetch updates. Accessibility and archival services can also make automated requests for legitimate reasons.

Useful bots usually identify themselves, publish documentation, respect access rules, and request pages at a rate the site can handle. None of those signals is impossible to fake, so identity still needs verification when it matters.

Bots that create cost or harm

Malicious automation includes credential stuffing, spam, inventory hoarding, vulnerability scanning, and content scraping that ignores stated rules. A bot can also be harmful without criminal intent if it makes so many requests that normal visitors suffer.

Some cases sit in the middle. An SEO crawler may be authorised for one customer and unwanted on another site. An AI crawler may respect robots.txt but still conflict with a publisher’s preferred use of its work.

Manage behaviour, not just names

Start with logs. Look at request rate, paths, failures, traffic patterns, and whether the visitor provides any value. Then choose a proportionate response:

• publish clear robots.txt rules for crawlers that honour them;
• rate-limit expensive or repetitive requests;
• require authentication for non-public data;
• use a firewall or bot-management service for abusive patterns; and
• block verified user agents or networks when their access has no value.

Do not use robots.txt to protect secrets. It is a public request, not a lock, and malicious bots can ignore it.

I would avoid blocking every automated request by default. Bots keep much of the web discoverable and connected. Allow the ones that behave well, measure the uncertain ones, and make the abusive ones somebody else’s log problem.

← Back to all articles