I've been archiving everything I've written since 2008 — chat logs, forum posts, AI-assistant sessions, the works. This year I turned that archive into a testbed for a question I couldn't stop thinking about: can a model learn to reply the way I reply? Not "write a poem in my style" — actually continue my real conversations, judged against what I actually said.
This page is the honest version: the pipeline, the evaluation harness, and the first real numbers — including the parts where the numbers say "not yet."
The corpus
| records | 33,184 |
| messages | 3,489,132 |
| platforms | 28 |
| date range | 2008 → 2026 |
| schema validation errors | 0 |
Every record carries W3C-PROV provenance (wasDerivedFrom,
wasGeneratedBy, generatedAtTime), so any derived number on this page can
be walked back to the original export it came from. Records are
content-addressed — the validator checks that unique IDs, unique content
hashes, and record count are all the same number, which is a cheap way to
prove there are no silent duplicates and no silent losses. The stats table
above isn't typed by hand; a script regenerates it from the validator output
and refuses to run if validation failed.
I'm not going to pretend the archive is perfect. 13 files an earlier pipeline catalogued are gone from disk, and 3 files were deleted by an over-broad cleanup rule before anything could stop it. The data card records all of it, because an archive that hides its own losses isn't an archive, it's marketing.
The pipeline, in words
raw exports (28 platforms, 70 source families)
→ one strict schema with provenance, content-addressed storage
→ hybrid retrieval index (BM25 + dense embeddings + reciprocal-rank
fusion + reranking)
→ scenario extraction: real conversations become (context → my actual
reply) pairs — 7,458 scenarios from one platform's 2026 messages
→ evaluation harness: an engine's drafted reply is scored against my
real reply on semantic similarity + a style fingerprint
→ training ladder: retrieval baseline → LoRA fine-tune → DPO
Two details in scenario extraction did the most work:
The split is by conversation, not by message. Train and eval scenarios never share a conversation, so nothing in the eval set can leak into what the engine learns from. Boring, load-bearing, easy to get wrong.
64% of the raw scenarios were duplicates. Multi-source exports duplicate whole conversations — the same exchange arrives via two different export paths and looks like two training examples. Content-hash dedupe dropped 13,126 of ~20.6k candidate scenarios. If I hadn't measured this, the eval would have been quietly grading the engine on memorized copies of its own training data. This is the kind of failure that never shows up unless you go looking for it.
The first honest numbers
Before training anything, you need to know what "good" even looks like. So the first engine is deliberately dumb: for each scenario, retrieve the most similar thing I've ever said and submit it verbatim. A copy of me, replying from the wrong conversation.
Scored on held-out scenarios against my real replies:
| metric | copy-baseline | read it as |
|---|---|---|
| semantic similarity (0–1) | 0.591 | "how close in meaning to what I actually said" |
| style fingerprint distance (0–1, lower = better) | 0.215 | "how far from my measured style stats" |
| length ratio | 0.48 | the retrieved messages were ~3.2× longer than my real replies |
What these numbers actually say:
- Even copying my own past messages only scores 0.59 on meaning. That's the finding. My replies are context-contingent — what I say depends on the specific conversation, not on having a stock of reusable lines. Context-contingency is the hard part, and it's exactly what retrieval alone can't solve. Any generative model I train has to clear this bar to be worth anything.
- The 3.2× length drift is a real failure. Retrieval favors long, information-dense past messages; my actual chat replies run short. An engine that sounds like me but says three times too much of it isn't me.
- The style distance (0.215) flatters the baseline twice. Once by construction — it outputs literal past messages, so style stats mostly self-match. And once by a leak I found in my own setup: the style corpus overlapped the eval time period, so the baseline could occasionally retrieve a message near-identical to an eval target. The fix (timestamp-filtering the corpus) is a required step before any trained model gets compared against this number. I'm publishing the flattered number anyway, labeled as such, because that's what the run produced.
- The harness also supports an LLM-judge A/B test ("which reply is the real person?"). On the first generative run it reported a 0.44 fooling rate — but I'm not headlining it: the judge shares a model family with the generator, the sample was small, and the literature is blunt that LLM judges are not evidence of real indistinguishability. It's an iteration-speed metric, nothing more.
If a number on this page ever looks impressive, reread it. 0.59 is a floor, not a result. The result is the harness: every future claim about this project has to beat a measured baseline on 1,550 scenarios from held-out real conversations, or it doesn't ship.
Privacy engineering
This project is only defensible if the boundary is absolute, so it's built in, not bolted on:
- Everything runs locally. Archive, indexes, training data, evaluation — one machine I own. No personal data leaves it. The archive itself is read-only to every tool that touches it; derived data is written elsewhere.
- Quarantine over deletion. Credentials and browser-profile material swept up by old exports are catalogued (so the record is complete) but flagged quarantined and excluded from anything published or processed — 531 assets currently sit in that state.
- What will never be published: the corpus, the message text, other people's words, the retrieval indexes, the scenario files, and any model checkpoint trained on them. What gets published is what you're reading: aggregate numbers, methods, and code that works on anyone's archive.
- Fine-tuning, when it happens, uses only my own turns as training targets, and anything that ever talks to another human gets labeled as automated. The eval bar is "indistinguishable"; the deployment rule is "disclosed."
What's next
The training ladder, each rung gated by the same harness:
- Generative baseline — a small local model, grounded with retrieved examples. First preliminary run (small 3B model, 25-scenario subset, CPU): semantic similarity 0.55, style distance 0.49 — it loses to the copy-baseline on both axes, which is exactly its job: an honest floor. The style breakdown is the interesting part — the model starts every message lowercase (I don't, only 44% of the time), never uses ellipses or emoji the way I do, and almost never reaches for my actual vocabulary. That's the gap fine-tuning exists to close, and now it's measured.
- LoRA fine-tune — low-rank adaptation on ~6k (context → reply) pairs, loss restricted to my turns. Gate: beat the generative baseline on semantic similarity and style distance, on the fixed eval set, after the style-corpus fix. If it can't, it doesn't graduate.
- DPO with my real replies as the preference signal — every real reply is preferred over every model attempt at the same context, so the training objective literally is "be less distinguishable from me." Stops the round it fails to move the eval.
I'll publish each rung's numbers here, wins and faceplants both. The archive took 18 years to accumulate and the harness took a lot longer than the model will — which is, I've come to accept, the actual lesson.