2.18 million observations into 117,820 trends
I built a small public-data product from official US and UK name records, with repeatable imports, explicit market boundaries, and immutable releases instead of live database rendering.
2,181,032 observations → 117,820 trends in under 4.5 seconds on the recorded PostgreSQL 17 run
How the pieces connect
- SSA and ONS source files
- Normalized observations
- Trend calculations
- Release validation
- Immutable Rust serving
The job
Official datasets arrive with different periods, formats, geographic coverage, and revision behavior. The public product still needs one coherent route and trend model.
The hard part
Preserve source meaning, avoid inventing comparability, and make every public release reproducible from frozen inputs.
How I built it
- Treat each country as its own market contract rather than blending incompatible datasets.
- Store normalized observations before deriving trends so calculations can be repeated and audited.
- Generate the full public release from frozen inputs and reject it when route or data checks fail.
- Serve the accepted output as immutable files with no public database dependency.
What I verified
- Official SSA and ONS ingestion
- 2,181,032 production observations
- 117,820 calculated trends
- Recorded transformation under 4.5 seconds on PostgreSQL 17 settings
Current state: The recorded build and both national-source pipelines are working. Traffic and commercial results are not part of this case.