The problem
Everything I have written online since 2008 was scattered across dead platforms, half-broken export formats, and folders of files with no dates and no context. Every platform exports differently, none of them agree on what a "message" is, and most exports silently drop the metadata that makes a record trustworthy: when it was written, where, and in what thread.
If you want a machine to be able to search and understand eighteen years of your own writing, "a folder of JSON files" is not an archive. It is a liability with timestamps.
Constraints
- Every record has to carry provenance: which export it came from, when it was captured, and how it was transformed on the way in. A record you cannot trace is a record you cannot trust.
- Platforms disagree about everything, so the archive needs one schema that all 28 of them normalize into — without losing what made each source different.
- Re-running ingestion must never duplicate. Exports overlap, and the same message arrives in three different exports of the same account.
- It has to be queryable at full scale on my own hardware, not a sample of it.
What I built
A pipeline that ingests raw platform exports and normalizes them into a single validated store:
- One schema, enforced. Every record is validated against a JSON Schema on ingest. The current archive holds 33,184 records containing 3.49 million messages from 28 platforms, with zero schema errors — not "mostly clean", zero.
- W3C-PROV provenance. Each record carries a provenance chain in the W3C-PROV model: the source export it derives from, the ingestion activity that transformed it, and the agent that ran it. You can walk any message back to the exact file it came from.
- Content-addressed deduplication. Records are stored by content hash, so overlapping exports collapse into one canonical copy and re-ingestion is idempotent by construction.
- Hive-partitioned Parquet as the analytical layer, so the whole corpus is queryable with DuckDB without a database server.
- Hybrid retrieval over the top: BM25 and dense embeddings fused with reciprocal rank fusion, then reranked — so a search hits both the exact-phrase matches and the ones that only match in meaning.
Outcome
Eighteen years of writing is now one queryable, verifiable corpus. Any record can prove where it came from, duplicates are structurally impossible, and the retrieval layer means the archive answers questions instead of just holding files.
The same discipline that archives a codebase's history — schemas, provenance, content addressing, validation gates — applied to a person's output. Most organizations do not keep their data to this standard. It turns out one person can.