# Keep the crawler claim separate from verification and access

First find the path behavior with the supplied hunt. Do not require a crawler-like User-Agent, and do not remove an event when a label or range check appears familiar.

Keep three distinct facts in the analyst view:

| Fact | Record | Meaning |
| --- | --- | --- |
| Client's claim | Exact User-Agent from the original request, including a missing value | Client-supplied text; a name match is a triage label |
| Verification | Direct peer, trusted proxy chain, vendor-specific method, result, and check time | Evidence for that identity check at that time |
| Behavior and access | Receiving server, method, original target/path, status, surrounding requests, and resource policy | What was requested and whether access should be allowed |

An optional label filter can prioritize a copy of the results for review. Keep the full behavior results available. A substring such as `GPTBot` in a header is not proof of impersonation, crawler ownership, maliciousness, or authorization. An absent label is not evidence that a request is safe. Cloud-provider address ownership is insufficient to identify the request's operator.

Use the claimed service's current verification procedure: [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), [OpenAI](https://developers.openai.com/api/docs/bots), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), or [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). Record the document/list retrieval time and do not apply today's range membership as historical proof. [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended) is a robots control token, not a separate HTTP crawler User-Agent.

Keep private resources protected even after a successful identity check. Test ordinary and crawler-labelled requests separately against an authorized protected test location. Review bot exceptions, rate exceptions, and access controls for a bypass. Check response evidence through the application owner; avoid copying cookies, query tokens, response secrets, or identifying operational logs into a shared report. The fixture addresses are documentation-only examples; the tests perform no DNS lookups, web requests, or access-control changes.
