We keep a running identity card per brand we track: what it is, how loud it is, where it shows up, what's been accumulating around it. The card is a document, and documents have a maintenance problem. Before every use, someone (or some script) rereads recent history and rebuilds the card. Rebuilds are slow, they get skipped under deadline, and two people rebuilding the same card from slightly different inputs get slightly different cards.
Stream processors have a primitive that dissolves this: keyed state. You declare an accumulator per key (the key here is the brand), and the framework routes every incoming event to its key's accumulator, applies your update logic, and keeps the result durable. The profile stops being a document you rebuild and becomes a value the stream maintains.
The update logic is nothing exotic. New mention arrives → increment the counts, update last-seen, bump the source tally. Maybe thirty lines. What the framework adds is everything around those thirty lines that's genuinely hard to hand-roll:
- Durability. State is checkpointed. The process crashes, restarts, and every brand's accumulator is exactly where it was. Hand-rolled version: a half-written JSON file. - Scale-out for free. A thousand brands means a thousand accumulators, partitioned across workers by key. Your update logic doesn't know or care. - A single writer per key. Two events for the same brand can't race each other. This is the property that kills the two-people-rebuild-differently problem — there is exactly one current card per brand, by construction.
The subtle design decision is what stays in state and what stays in documents. Numbers, tallies, last-seen timestamps: state. Prose, analysis, judgment: documents, in the knowledge base, written by people (or at least reviewed by them). The stream processor snapshots its accumulators out as cards on a schedule, and the flow is strictly one-way. The moment documents feed back into state you have two systems of record and a drift problem, which is the disease you were curing.
What I like most about this framing is that it makes freshness a non-event. Nobody asks whether the profile is up to date, because "up to date" is the only condition the architecture can produce. The question shifts to a better one: is the update logic right? And that logic is thirty reviewable lines in one place, instead of rebuild code scattered across every consumer.
What's still open
Accumulators are additive by nature, and brands change. When a company rebrands or pivots, how does accumulated state age out? Time-decayed tallies, windowed sub-profiles, or an explicit reset-with-archive are all defensible. Picking one is a modeling decision about brand identity, and I don't think it has a purely technical answer.
Related
-