How to Block AI Crawlers

Back to Bots

How do you block AI crawlers?

Decide which crawlers may access which pages. Publish those preferences in robots.txt, apply the required WAF or bot rules, then inspect the results in request logs. Give each rule an owner, a reason and a rollback plan.

Step 1: Review known crawlers before choosing a policy

Build an inventory from current operator documents and your logs. Record the crawler name, purpose, source check, permitted paths and intended action. The detection guide explains how to find requests and check their claimed source.

OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot and Perplexity-User. Make a separate decision for each purpose.

Step 2: Publish a narrow robots.txt policy

For example, a site that wants to restrict OpenAI and Anthropic training crawls can add these groups to the robots.txt file at the root of each relevant host:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Merge the groups with your current file and review any existing rules for search and retrieval bots. Check that the public robots.txt URL returns the intended file successfully.

Robots.txt relies on crawler cooperation. OpenAI says robots.txt rules may not apply to user-triggered ChatGPT-User requests. Perplexity says its user-triggered fetcher generally ignores those rules. Anthropic documents robots.txt support for its robots. Enforce private access with authentication and server-side authorization.

For Google's content-use preference, review Google-Extended. It has no separate HTTP user-agent string to block. Apple documents the Applebot-Extended content-use setting in About Applebot. Check these settings separately from rules that stop the underlying crawler fetching pages.

Step 3: Apply access rules to the intended routes

Before adding a rule, save the current configuration and record its position among existing WAF, bot and rate-limit rules. Start with one host and a small group of paths. Use logging or a staging environment to review matches before blocking production traffic.

For an approved crawler, combine its claimed name with the operator's documented source checks where available. Keep source lists current. An allow exception should cover the required paths and controls; retain authentication and other protections needed by the application.

For unwanted traffic, match the affected paths and the request details available in your system. User-agent matches can catch clients that identify themselves. Include request behavior in the review because clients can change that header.

Interactive browser challenges can prevent automated crawlers from completing a fetch. Test the chosen action with the clients you intend to allow. Review rule order when an approved crawler still receives a block or challenge.

Step 4: Set a measurable rate limit

Choose a path group, counting key, interval, threshold and action. Record why the threshold fits the route's capacity. For example, assess product search separately from cached articles because the origin work can differ.

Peakhour documents named stateful zones keyed by IP, country, ASN, header, cookie, TLS fingerprint or HTTP/2 fingerprint in its rate-limiting guide. It also documents counting selected responses before limiting later requests. Confirm available controls for your deployment.

Check how the key groups clients. IP rotation can spread requests across counters. A shared fingerprint can put valid clients in the same counter. Select a threshold from observed traffic and review affected clients before enforcing it. A cookie or header supplied by the client needs a trust check before it can identify a client reliably.

Step 5: Inspect results in logs

Keep the change time, rule ID, path scope and intended outcome with your review notes. Compare equivalent periods before and after the change, allowing for normal traffic variation.

Check CDN and security logs alongside origin logs. Cached responses and edge blocks can be absent from origin logs. Inspect a response body when a successful status may contain a challenge. A request you send with a crawler user-agent can test a header match; its source check still reflects your own address.

Step 6: Keep the policy current

Review new crawler names, operator source lists, approved exceptions and changes in request volume. Recheck robots.txt after application or CDN changes. Reverse a rule if it blocks required access, then narrow the match and test again.

For a provider evaluation, use the security evaluation checklist to agree on client coverage, logs, ownership and rollback. See Peakhour bot management for documented request actions and how to detect AI crawlers for source checks. The OpenAI, Anthropic and Perplexity primary documents were checked on 8 November 2026. Revisit operator guidance when updating rules.