How to Detect AI Crawlers

Back to Bots

How do you detect AI crawlers?

Start with requests in your CDN, security and origin logs. Record the claimed crawler name, check its source using the operator's current documents, then review the paths and actions in your logs. Keep the claimed name and the verification result in separate fields.

Step 1: Review known crawler names and purposes

Use the operator's current documentation to maintain a crawler list. Record its purpose, verification method, source URL, last review date and policy owner.

These names describe different uses. A site can choose to allow search indexing and user-requested retrieval while restricting training crawls. Review each name separately before changing policy.

Google-Extended is a robots.txt product token with no separate HTTP user-agent string. Google's existing crawlers fetch the pages. See Google's common crawler reference. Apple describes Applebot-Extended as a control over use of content collected by Applebot in About Applebot. Record those preferences in your policy inventory; search request logs for the documented fetching crawler.

Step 2: Find candidate requests in logs

For a text access log that includes the User-Agent field, start with:

rg -i 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log

The matches show requests claiming those names. A client can send any user-agent string. Also review high-volume requests with unfamiliar names or ordinary browser headers.

In structured logs, select the same names in the user-agent field. Group by host, path, method, source IP, status, action and rule ID. Count requests and bytes for a fixed period, such as the previous 24 hours. Check CDN logs for requests served from cache or blocked before origin.

Step 3: Check the source against operator guidance

OpenAI publishes separate IP range lists for its documented bots. Perplexity also publishes separate lists for its crawler and user fetcher. Follow each operator's linked lists and refresh them when they change. Anthropic's robot documentation links to its crawler source list. Record the check date and whether the source matched.

Use the visitor address supplied by your trusted edge. When a CDN forwards traffic, confirm which client-IP header it sets and which proxies your origin trusts. A freely supplied forwarding header can give a false source address.

If an operator documents DNS verification, follow the full procedure, including checking that forward DNS resolves back to the original IP. Record an unknown result when the available checks cannot establish the source.

TLS and HTTP/2 fingerprints, request timing and proxy information can help find automation or inconsistent behavior. A shared fingerprint can group many clients, so review its effect on valid traffic before using it in a rule. These checks describe client behavior; operator verification establishes the claimed source.

Step 4: Inspect paths, actions and outcomes

For each crawler group, review:

• Public articles and product pages requested, including query parameters and crawl depth.
• Requests to costly search routes, APIs, login, checkout or account pages.
• Request rate, bytes transferred, origin response time and repeated errors.
• Allow, log, rate-limit, challenge and block actions, with the matching rule ID.
• Requests that reached origin and requests handled by the CDN or security service.

Keep a small review table in your own logs or reporting tools:

Use your measured counts. Keep credentials, customer identifiers and private query values out of shared examples.

Step 5: Choose a policy and review its results

Give each crawler purpose an owner and a decision: allow approved public access, set a route limit, or block disallowed requests. Record exceptions and a review date. Follow how to block AI crawlers to apply the choice.

After a change, compare equivalent log periods. Check whether approved crawlers still fetch the intended pages, which rule explains each denial, and whether origin load or errors changed. A 200 response can contain a challenge page; inspect the action and returned content. Save the previous rule so you can reverse an unwanted change.

For a provider trial, use the security evaluation checklist to agree on routes, client types, logging and rollback. Peakhour's bot management page describes request policy and logs. Confirm the fields and controls available in your deployment.

For more names to review, see AI crawler user agents. The OpenAI, Anthropic and Perplexity primary documents were checked on 8 November 2026. Revisit operator guidance when updating rules.