A crawler user agent is a claim, not an identity
Edge logs can show that requests occurred and that a user agent claimed a search, product, or AI crawler family. Provider identity requires another step: obtain the source IP from a trusted origin path and validate it using that provider’s documented method.
HTTP requests, not visitors · family classification before provider confirmation · trusted origin IP required
Count the thing the server actually observed
An edge or origin log records HTTP requests. One crawler can request the same URL repeatedly, fetch several resources, retry after an error, or operate through more than one agent family. The total is therefore a request count, not a visitor count and not a count of unique crawlers.
Active days and requested routes can describe crawl attention. They cannot prove that a human visited, that a page was indexed, or that a provider used the response in a search or AI product.
Normalize the claim into a provider family
A user-agent parser can group tokens into useful families such as search crawler, product crawler, AI training agent, or AI answer agent. That normalization helps compare behavior without treating every user-agent version as a new actor.
The family label still comes from client-supplied text. User agents are easy to copy, so the visible label must remain “user-agent-claimed” until network identity is checked.
- Retain the original observation separately from the normalized family
- Version family rules so historical totals can be interpreted
- Keep unknown and contradictory agents visible
- Do not silently promote a familiar brand token to provider-confirmed
Trust an origin IP only through a trusted path
If the origin is locked behind a known proxy or CDN, use the source field that proxy sets and strips. If the public can reach the origin directly, a forwarded IP header can be forged and must not be treated as identity evidence.
After the trustworthy source IP is established, apply the provider’s documented verification method. That may be a published IP range or a reverse-DNS check followed by forward confirmation. Record which rule and observation time produced the result because provider ranges and hostnames can change.
Store confidence, not one overloaded provider field
A useful record separates the observed request, claimed user-agent family, source-IP provenance, verification method, and final confidence state. The output can then say provider-confirmed, user-agent-claimed, unknown, or contradictory without destroying the raw evidence.
The current public analytics claims stop at user-agent-derived families unless source-IP validation has been performed. The verification process is a requirement for promotion, not a claim that every stored request has already passed it.
Use crawler data for operational questions
Request totals can show which routes receive crawler attention, whether a deployment changed fetch patterns, and when an agent family was last active. Combined with Search Console, Bing, index checks, and release records, they help an operator decide what to inspect next.
They still do not prove indexation, citation, ranking, discoverability, or audience. Those outcomes require their own sources and their own declared limitations.
What the implementation proves
- Working edge-crawler observations inside a multi-property self-hosted analytics system
- Provider-family reporting explicitly qualified as user-agent-claimed
- Separate Google Search Console, Bing, operator-triggered index-count, and crawler views
- Permanent crawler rollups retained before bounded raw-data cleanup
Observed request first. Provider identity last.
Each stage adds evidence without rewriting the previous one. Failed or unavailable checks remain visible as claimed, unknown, or contradictory.
Observation, interpretation, and limit.
Every label states what was measured and what additional evidence would be required to make a stronger claim.
| Record | Implemented use | Boundary |
|---|---|---|
| HTTP request | The server or trusted edge observed a request at a route and time. | It is not a human visit, a unique crawler, an indexed page, or a product citation. |
| Claimed family | The user-agent token is normalized into a versioned crawler or agent family. | Client-supplied user-agent text is spoofable and remains a claim. |
| Origin source IP | The address comes from the connection or a header controlled by a trusted, origin-locked proxy. | A public-origin forwarded header is not trustworthy identity evidence. |
| Provider confirmation | The source address passes the provider’s documented IP-range or reverse-and-forward-DNS procedure. | The result is tied to that provider rule and observation time; it does not validate unrelated requests. |
| Aggregate | Requests can be grouped by family, route, and active day for operational comparison. | Aggregation does not turn requests into visitors or prove indexing, ranking, citation, or demand. |
Limits: Current provider-family totals remain user-agent-claimed unless origin source IPs were validated. · HTTP requests are not visitors, unique people, unique crawler processes, index counts, or citations. · Provider verification methods differ and can change; the method and observation time belong in the evidence.