crawlreplay: test crawler policy and candidate releases

The job

A robots.txt edit or deployment can look correct in a synthetic checklist while changing what happens to URLs that crawlers already request. Access logs contain the concrete request history needed to test that risk before release.

The hard part

Replay useful request evidence without forwarding captured secrets, trusting an arbitrary log host as a network destination, treating a user-agent string as verified identity, or letting one run consume unbounded network or response resources.

How I built it

1. Keep robots-policy comparison and candidate-release comparison as separate commands with separate inputs and failure thresholds.
2. Require explicit baseline and candidate origins, empty captured query values by default, and never replay captured cookies, authorization, bodies, or client headers.
3. Treat recognized crawler user agents as claims unless their source addresses match the bundled vendor CIDR snapshots for the supported Google or Bing product.
4. Constrain redirects to the exact allowed origins, reject special-use network addresses, disable ambient HTTP proxies, and bound concurrency, starts, redirects, time, and response bytes.
5. Keep reports suitable for people and CI with redacted query values, versioned JSON, SARIF 2.1.0 output, and documented exit codes.

What I verified

• Policy checks compare old and proposed robots.txt rules against URLs in real access logs
• Release checks compare status, redirects, canonical links, indexability, hreflang, schema types, and content fingerprints
• Nginx combined and JSON logs, Cloudflare Logpush HTTP-request logs, gzip input, multiple files, and standard input
• Terminal, versioned JSON, Markdown, and SARIF 2.1.0 report formats
• Apache-2.0 source, locked Rust workspace, composite GitHub Action, and digest-pinned non-root Docker image