Deception Check

FIELD RESEARCH / CRAWLER CLAIMS AND PRIVATE FILES

A crawler name is not a credential.

The visitor's introduction should never decide which files your website will serve.

A request can introduce itself as a familiar crawler and still ask for a private configuration file. Read the request before trusting the label.

In our retained research-decoy records, 17 recorded source addresses used at least five AI or search-crawler labels each. That group sent 7,771 requests, including 1,723 with .env in the requested path. Environment files often hold application settings and credentials. They deserve the same protection regardless of the visitor's name.

Our internet-facing decoys imitate services so we can study the requests visitors send. The useful response is practical: keep the claimed identity, source verification, and requested resource together. Then check whether crawler exceptions can change the access decision.

The introduction kept changing. The file requests remained.

Our selected export contains 16,395 crawler-labelled requests from September 8 through October 7, 2026, UTC. Thirteen of the multi-label addresses each used thirteen different labels. These are client-supplied User-Agent strings, the part of an HTTP request where software usually identifies itself.

SELECTED CRAWLER-LABELLED REQUESTS / SEPTEMBER 8 TO OCTOBER 7, 2026, UTC

17 recorded addresses accounted for 47.4% of these requests.

7,771 requests
17 addresses, five or more labels each

8,624 requests
All other requests in the selected export

1,723 requests from the multi-label group had .env in the path.

Counts describe requests and recorded addresses, not authenticated organizations or people. The 1,723 requests are a subset of the 7,771, not an additional category.

Across the full selected export, requests also included /.git/, application-management paths such as /actuator, and URLs containing a cloud-metadata destination. Those are different investigation leads. A path naming a private file establishes a request, not that the file existed or was returned.

Keep three checks in the same view

ONE REQUEST / THREE SEPARATE QUESTIONS

  1. 01 / CLAIM

    What did it call itself?

    Retain the original User-Agent and request identity.

  2. 02 / VERIFY

    How did you check the source?

    Use the named provider's method. Record the result and check time.

  3. 03 / BEHAVIOR

    What did it ask for?

    Keep the path, method, response, and nearby requests.

Resource permission stays separate.

A verified crawler still does not get permission to read a private file.

This is an investigation workflow. Identity verification and authorization answer different questions.

Claim. A User-Agent is supplied by the client. Keep it as evidence of what the visitor said, rather than treating it as a login.

Verification. Start with the connection source recorded by a trusted collection point. Behind a proxy or content-delivery network, use its documented client-address field and trust settings. A client-controlled forwarding header is not a reliable substitute.

Behavior. Review the resource and its response. A public article and an environment file need different access policies. A status of 200 can be a catch-all page, so inspect a safely retained response reference or content classification before concluding that a secret was disclosed.

Use the verification method for the specific crawler

Current provider guidance, reviewed November 5, 2026
ProviderWhat to checkKeep with the result
GoogleUse the appropriate crawler/fetcher IP list or its documented reverse-DNS check, including forward resolution back to the source address.Crawler category, method, source, and time.
OpenAICheck the separately published ranges for GPTBot, OAI-SearchBot, or ChatGPT-User, as appropriate.The particular bot list and dated snapshot.
PerplexityUse its bot-specific published guidance and address lists.The claimed bot, list, and time.
AnthropicUse its crawler guidance and linked crawler address list, rather than unrelated API egress ranges.The crawler list and verification time.

Record a result such as matched, not matched, or unknown, together with what was actually checked. Current range membership does not prove who operated a historical request. Google-Extended is a robots.txt control token, not a separate HTTP User-Agent identity.

Test whether the label changes the boundary

Choose an approved test resource that should remain private and contains no real credentials. Send an ordinary request, then repeat it with only the User-Agent changed to a crawler label. Both requests should leave the protected content inaccessible. Test approved crawler access to public content separately.

AUTHORIZED TEST / ORDINARY HEADER

Private test resource

Record the access decision and a safe response reference.

Expected: no protected content returned

AUTHORIZED TEST / CRAWLER LABEL

The same private resource

Change only the label. Retain the new request and response.

Expected: no protected content returned

Review any bot-management, rate-limit, WAF, or cache exception with the web owner. A web application firewall, or WAF, inspects web traffic; an exception intended to reduce crawler friction should not bypass the controls protecting private resources.

Finally, check the analyst's view. Can someone see the requested resource and response, or only a reassuring bot category? Group related requests into a useful investigation while retaining the original events.

TAKE ONE APPLICATION THROUGH THE CHECK

A worksheet for the web owner and responder

Record the collection point, verification source, protected test resource, both responses, and the next action. Keep real credentials and sensitive response bodies in restricted evidence storage.

Download the two-page worksheetOpen the readable worksheet

Hunt for the resource request first

The detection companion starts with ordinary web-access telemetry and a standard Sigma rule. It looks for selected private-looking configuration paths without requiring a crawler label or a successful response. That keeps requests visible when the header changes.

Use the claimed crawler name as triage context after finding the request. These rules identify probes for review; they do not verify a provider, prove a secret was returned, or establish compromise.

Download the hunt companion ZIP or read the quickstart. Start with the source you already collect.

Standard Sigma: selected configuration requests

Map the real method to cs-method and the path without its query to cs-uri-stem. This standard webserver rule covers GET and HEAD, regardless of User-Agent or response status. It matches the eight listed endings literally and case-insensitively, including below a deployment prefix.

Complete Sigma rule
title: Request for Selected Private Configuration Paths
id: 240846aa-01c3-4633-89d1-a61c1f046175
status: experimental
description: Detects GET or HEAD requests for selected configuration and management path endings, regardless of claimed User-Agent. Review access policy and response evidence.
author: Deception Check
date: 2026-11-05
references:
    - https://sigmahq.io/sigma-specification/specification/sigma-appendix-taxonomy.html
    - https://docs.spring.io/spring-boot/reference/actuator/endpoints.html
logsource:
    category: webserver
    definition: Requires a path-only cs-uri-stem and cs-method from inbound web access logs. No query string or decoding is performed by this rule.
detection:
    selection_method:
        cs-method:
            - GET
            - HEAD
    selection_path:
        cs-uri-stem|startswith: '/'
        cs-uri-stem|endswith:
            - '/.env'
            - '/.env.local'
            - '/.env.production'
            - '/.git/config'
            - '/.git/head'
            - '/actuator/env'
            - '/actuator/configprops'
            - '/wp-config.php'
    condition: selection_method and selection_path
fields:
    - c-ip
    - cs-method
    - cs-uri-stem
    - sc-status
    - cs-user-agent
falsepositives:
    - Authorized security testing or monitoring
    - Intentionally public configuration or management endpoints
level: medium

Choose the native hunt for your access logs

Run a source-only preview of about 20 events first. Each guide includes the selector, required data, and a practical first test. Temporary hunt_ fields are created inside the query; they are not a custom ingestion schema.

Microsoft Sentinel / Azure Monitor IIS

IIS W3C logs collected by Azure Monitor Agent into native W3CIISLog; workspace query permission and ingestion entitlement. The 24-hour hunt returns the original path, recorded source, raw User-Agent, status, and receiving server. This is an IIS access-log query, not an MDE endpoint rule.

Source quickstart and preview · Download query

// Native IIS access logs in Azure Monitor / Sentinel; see README.md.
W3CIISLog
| where TimeGenerated >= ago(24h)
| where csMethod in~ ("GET", "HEAD")
| where isnotempty(csUriStem) and csUriStem startswith "/"
| extend hunt_original_path = csUriStem
| extend hunt_raw_path = tostring(split(hunt_original_path, "?")[0])
| extend hunt_path = tolower(url_decode(hunt_raw_path))
| where hunt_path endswith "/.env"
    or hunt_path endswith "/.env.local"
    or hunt_path endswith "/.env.production"
    or hunt_path endswith "/.git/config"
    or hunt_path endswith "/.git/head"
    or hunt_path endswith "/actuator/env"
    or hunt_path endswith "/actuator/configprops"
    or hunt_path endswith "/wp-config.php"
| project TimeGenerated, Computer, sSiteName, csHost, cIP, csMethod,
    hunt_original_path, hunt_path, csUserAgent, scStatus, scBytes
| order by TimeGenerated desc
Splunk combined Nginx access logs

Nginx combined-format access lines collected as access_combined; replace YOUR_WEB_INDEX. Requires search and ingestion entitlement, with no Enterprise Security or accelerated model requirement. The query parses the original request line and retains _raw.

Source quickstart and preview · Download query

index=YOUR_WEB_INDEX sourcetype=access_combined earliest=-24h
| rex field=_raw "^(?<hunt_client>[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?<hunt_method>[^ ]+) (?<hunt_target>[^ \"]+) HTTP/[0-9.]+\""
| rex field=_raw "^(?:[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?:[^ ]+) (?:[^ \"]+) HTTP/[0-9.]+\" (?<hunt_status>[0-9]{3})(?: |$)"
| where isnotnull(hunt_target) AND (upper(hunt_method)="GET" OR upper(hunt_method)="HEAD")
| where like(hunt_target,"/%")
| eval hunt_raw_path=mvindex(split(hunt_target,"?"),0)
| eval hunt_path=lower(urldecode(hunt_raw_path))
| where like(hunt_path,"%/.env")
    OR like(hunt_path,"%/.env.local")
    OR like(hunt_path,"%/.env.production")
    OR like(hunt_path,"%/.git/config")
    OR like(hunt_path,"%/.git/head")
    OR like(hunt_path,"%/actuator/env")
    OR like(hunt_path,"%/actuator/configprops")
    OR like(hunt_path,"%/wp-config.php")
| table _time host source hunt_client hunt_method hunt_target hunt_path hunt_status _raw
| sort 0 - _time
Sumo Logic Nginx combined logs

Collected Nginx combined access lines in Log Search; replace YOUR_NGINX_ACCESS_CATEGORY and select a short time window. Requires normal log entitlement and permission to search that source. The query parses _raw at search time.

Source quickstart and preview · Download query

_sourceCategory="YOUR_NGINX_ACCESS_CATEGORY"
| parse regex "^(?<hunt_client>[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?<hunt_method>[^ ]+) (?<hunt_target>[^ \"]+) HTTP/[0-9.]+\""
| parse regex "^(?:[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?:[^ ]+) (?:[^ \"]+) HTTP/[0-9.]+\" (?<hunt_status>[0-9]{3})(?: |$)" nodrop
| where toUpperCase(hunt_method) = "GET" or toUpperCase(hunt_method) = "HEAD"
| parse regex field=hunt_target "^(?<hunt_raw_path>/[^?]*)"
| urldecode(hunt_raw_path) as hunt_decoded_path
| toLowerCase(hunt_decoded_path) as hunt_path
| where hunt_path matches "*/.env"
    or hunt_path matches "*/.env.local"
    or hunt_path matches "*/.env.production"
    or hunt_path matches "*/.git/config"
    or hunt_path matches "*/.git/head"
    or hunt_path matches "*/actuator/env"
    or hunt_path matches "*/actuator/configprops"
    or hunt_path matches "*/wp-config.php"
Elastic Nginx ES|QL decoded

Elastic Nginx integration access streams, ECS method and single-valued url.original, plus ES|QL/read permission. Targets 9.2+ and decodes the original path once. Choose a Discover time window; narrow it if results reach the 10,000-row cap.

Source quickstart and preview · Download query

FROM logs-nginx.access-*
| WHERE event.dataset == "nginx.access"
| WHERE TO_UPPER(http.request.method) IN ("GET", "HEAD")
| EVAL hunt_original_target = url.original
| WHERE hunt_original_target IS NOT NULL
| EVAL hunt_raw_path = MV_FIRST(SPLIT(hunt_original_target, "?"))
| WHERE STARTS_WITH(hunt_raw_path, "/")
| EVAL hunt_path = TO_LOWER(URL_DECODE(hunt_raw_path))
| WHERE ENDS_WITH(hunt_path, "/.env")
    OR ENDS_WITH(hunt_path, "/.env.local")
    OR ENDS_WITH(hunt_path, "/.env.production")
    OR ENDS_WITH(hunt_path, "/.git/config")
    OR ENDS_WITH(hunt_path, "/.git/head")
    OR ENDS_WITH(hunt_path, "/actuator/env")
    OR ENDS_WITH(hunt_path, "/actuator/configprops")
    OR ENDS_WITH(hunt_path, "/wp-config.php")
| LIMIT 10000
Elastic Nginx ES|QL literal

Elastic Nginx integration and ECS url.path / http.request.method, with ES|QL/read permission. Targets 8.14+; matches the ingested path with no added decode. This fallback does not promise encoded-path coverage.

Source quickstart and preview · Download query

FROM logs-nginx.access-*
| WHERE event.dataset == "nginx.access"
| WHERE TO_UPPER(http.request.method) IN ("GET", "HEAD")
| EVAL hunt_path = TO_LOWER(url.path)
| WHERE STARTS_WITH(hunt_path, "/")
| WHERE ENDS_WITH(hunt_path, "/.env")
    OR ENDS_WITH(hunt_path, "/.env.local")
    OR ENDS_WITH(hunt_path, "/.env.production")
    OR ENDS_WITH(hunt_path, "/.git/config")
    OR ENDS_WITH(hunt_path, "/.git/head")
    OR ENDS_WITH(hunt_path, "/actuator/env")
    OR ENDS_WITH(hunt_path, "/actuator/configprops")
    OR ENDS_WITH(hunt_path, "/wp-config.php")
| LIMIT 10000
CrowdStrike CQL Apache access logs

Separately ingested Apache common/combined access logs with preserved @rawstring in Falcon LogScale or Next-Gen SIEM, compatible CQL functions, search access, and third-party ingestion entitlement. Verify the integration access-log tag. A Falcon endpoint sensor alone is insufficient.

Source quickstart and preview · Download query

#logtype="apache-access-log" @rawstring=*
| regex("^(?<hunt_client>[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?<hunt_method>[^ ]+) (?<hunt_target>[^ \"]+) HTTP/[0-9.]+\"", field=@rawstring, flags="", strict=true)
| hunt_target=*
| regex("^(?:[^ ]+) [^ ]+ [^ ]+ \\[[^\\]]+\\] \"(?:[^ ]+) (?:[^ \"]+) HTTP/[0-9.]+\" (?<hunt_status>[0-9]{3})(?: |$)", field=@rawstring, flags="", strict=false)
| upper(field=hunt_method, as=hunt_method_upper, locale="en")
| in(field=hunt_method_upper, values=["GET", "HEAD"])
| regex("^(?<hunt_raw_path>/[^?]*)", field=hunt_target, flags="", strict=true)
| hunt_raw_path=*
| urlDecode(field=hunt_raw_path, as=hunt_decoded_path)
| hunt_decoded_path=*
| lower(field=hunt_decoded_path, as=hunt_path, locale="en")
| case {
    text:endsWith(string=hunt_path, substring="/.env") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/.env.local") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/.env.production") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/.git/config") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/.git/head") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/actuator/env") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/actuator/configprops") | hunt_finding := "selected configuration-path request";
    text:endsWith(string=hunt_path, substring="/wp-config.php") | hunt_finding := "selected configuration-path request";
}

Compatibility and validation: the decoded hunts split at the first literal ? before one decode pass; Sigma and the Elastic literal fallback add no decoder. Query-only names, other methods, backup endings, trailing slashes, directory-only requests, and double encoding are outside this selected coverage. The list is narrower than the research counts. See the shared source, decoding, and compatibility notes, then test your actual ingestion.

Safe local fixtures check the distributed predicates and request-parser expressions. Native compilation, vendor/customer-tenant execution, ingestion execution, Sigma backend conversion, alert configuration, and performance are NOT RUN. These are experimental hunt starters. No suitable stock inbound request-path source has been established here for an endpoint-only Sophos or MDE translation.

Keep the original event access-controlled. A client address can be a proxy; use only trusted forwarded-source handling before identity verification or grouping. The triage workflow keeps the client-supplied crawler claim, vendor verification, resource permission, and response evidence separate.

Make the exception explainable

For one application, ask the web owner to show the protected resource, the crawler verification method, and the exception that applies. Then ask the responder to find both authorized test requests and explain what the server returned.

Would your crawler exception pass that check? The answer should come from the access decision and evidence, rather than the name in the header.

Sources and method

The research counts use retained decoy records from September 8 through October 7, 2026, UTC, with identical full exported rows collapsed. They were independently recounted on November 5. The export tracks 15 client-supplied AI/search-crawler labels. Recorded source addresses come from request records or associated connection sessions; one address can represent multiple users or services.

The multi-label group consists of the 17 recorded addresses with at least five different labels. Its 7,771 requests are 47.4% of the 16,395 selected requests. Thirteen of those addresses each used thirteen labels. The 1,723 count is a case-insensitive .env substring count in that group's exported request paths. It is not the detection pack's narrower selected-path test.

Current verification guidance was refreshed on November 5. The historical request counts do not authenticate the providers named in the headers. Requests, provider verification, resource permission, returned contents, and later credential use remain separate observations.

Read the counting method and download the deidentified aggregate counts. The related AI-tool file-hunting field note follows the deployment artifact and actual response. Provider verification sources are linked above.