Five public data products, one evidence pipeline
I built one backend that can turn very different public sources into focused data products without pretending every vertical has the same facts, quality rules, or refresh cycle.
5 source-backed product configurations · exact 50,001-page snapshot-loader test
How the pieces connect
- Official or licensed sources
- Entities, facts, and source history
- Quality checks and publication rules
- Static product release
- Monitoring and rollback
The job
A useful public-data site needs more than a page generator. Source history, conflicting facts, freshness, publication rules, URLs, releases, and corrections all have to agree.
The hard part
Reuse the machinery without flattening vehicle recalls, public spending, software comparisons, travel plans, and risk records into one weak schema.
How I built it
- Keep source evidence and normalized facts separate so a public claim can be traced back to its origin.
- Give each product its own publication policy, route model, freshness rules, and presentation instead of hiding differences behind generic templates.
- Build immutable public snapshots and validate the complete route set before activation.
- Retain the previous release and the source ledger so bad data can be corrected without improvising in production.
What I verified
- Five distinct source-backed product packs
- Exact 50,001-page snapshot-loader test
- Per-fact provenance and conflict handling
- Release validation, activation, and rollback contracts
Current state: The five products, source tracking, quality checks, releases, and rollback path are built. Production cutovers and traffic data are still pending.