01 / PREPARE AN AUTHORIZED CHECK
Does a crawler label change
your access decision?
Choose a protected resource and compare two unauthenticated requests. Change only the User-Agent header. Both should remain denied.
Print this worksheet or keep the answers in your own case record. This page does not submit or save notes.
Name the test
System / application and environment
Authorized owner / scope and test window
Protected path or route reference
Expected deny behavior / policy reference
Keep the comparison fair
Ordinary User-Agent value / request reference
Claimed crawler value / request reference
Same method, path, network, and no signed-in session
CDN / cache / access-control owner
Use an inert test resource with the same protection policy. Use no real secret. Obtain approval for the target and timing; keep the method, path, cookies, and authorization state identical.
Inspect the response body as well as the status. Follow redirects and compare any cache behavior within the approved scope.
Separate identity from permission
- Claim: record the header exactly as received. It is the visitor's assertion.
- Verify: use the named operator's documented source-verification method. Record method, time, and result; do not treat a label as verification.
- Behavior: preserve the requested path, method, and response evidence.
- Permission: apply the resource's access policy regardless of crawler identity.
02 / RECORD THE RESULT AND ASSIGN THE DECISION
Keep the evidence.
Name the next owner.
Record references to restricted evidence. Keep raw source addresses, sensitive response bodies, and credentials out of shared copies.
| Comparison | Ordinary header | Claimed crawler header |
|---|---|---|
| UTC time / method | ||
| Claim / source reference | ||
| Verification method, time, result | ||
| Status / redirect / cache | ||
| Body reviewed / evidence reference | ||
| Expected vs. observed decision |
What does this result support?
Both denied: record the tested route, conditions, and evidence. This checks that boundary under those conditions.
Unexpected access or different handling: preserve restricted evidence, assign the access-control or cache owner, and retest after the correction.
Identity unresolved: keep it unresolved. Do not infer identity from a successful status or a familiar crawler name.
Close the loop
Decision / owner / due date
Retest time, result, and reviewer
Keep the record useful
Restricted case / evidence reference
Next review trigger / policy owner
