"Exactly-once processing" is the most engineering-flavored phrase in the streaming world. It means each event affects your results exactly one time, even across crashes and restarts — no double counting when a process retries, no silent gaps when one dies mid-write. Distributed-systems people treat it as a hard-won correctness guarantee, which it is.
I want to make a different case for it: exactly-once is an audit feature, and intelligence work is audit work.
An intelligence report is a chain of claims. The score rests on claims, claims rest on evidence, evidence rests on sources. We build reports so that every figure traces down that chain — a reader who challenges a number should land on the observations behind it. Here's the uncomfortable part: the chain is only as good as the record of observations at the bottom, and batch collection quietly corrodes that record. A collector that crashes and reruns writes some mentions twice. A run that half-fails drops a day of evidence, and nothing downstream knows. The published trend line is then resting on a record that's subtly wrong, in ways the totals don't reveal.
The streaming fix is an evidence journal: every observation flows through the stream processor into an append-only log, under exactly-once delivery. The processor checkpoints its progress, and writes commit atomically with the checkpoint. Crash, restart, replay — the journal ends up with each observed event once. Not approximately once. Once.
What you buy with that guarantee is replayability, and replayability is the whole audit posture:
- "Why does this score say 62?" becomes a bounded query: here is every journal row for that brand up to the timestamp stamped on the score. Recompute and you get 62. - "What did we know on 2026-06-15?" is answerable precisely, because the journal records when each item was discovered, separately from when it happened. - A methodology change can be applied retroactively by replaying the journal through the new logic, producing a clean restatement instead of a "figures before this date used the old method" footnote.
None of this requires the fancy math to be streaming. The scores, the graphs, the reports can all stay batch. The journal is the piece that wants a stream processor underneath it, because ingestion is where duplicates and gaps are born, and exactly-once at ingestion is precisely the disease control.
The unglamorous parts carry a lot of the value here. Monitoring vendors demo the real-time dashboard; the append-only log with a delivery guarantee is what lets you sit across from a skeptical client (or a regulator, or your own future self) and rebuild any number you've ever published. Instruments get trusted on exactly that.
What's still open
Retention policy is a real decision: an audit journal should outlive the reports that cite it, which implies retention measured in years and a storage-format choice you won't regret (boring, columnar, queryable by anything). And the guarantee has a boundary — exactly-once protects the path from processor to journal, while a collector that submits the same fact twice under two IDs still needs dedup discipline at the front door. The guarantee is strong; it isn't magic.
Related
-