I cloned Apache Flink this week and read it with one question in mind: what does a stream processor add to an intelligence stack, and what does it replace? This post is the answer I ended up with, and it doubles as a map of the series.
The stack, and the boundary running through it
Strip any serious intelligence operation (ours included) to its layers and you get four:
1. Collection. Pollers, scrapers, transcript feeds. Evidence comes in. 2. Analysis. Structure gets found: knowledge graphs, gap detection, scoring against a rubric. 3. State. What we currently believe: brand profiles, rankings, the knowledge base. 4. Composition. The report: argument, evidence, editorial craft, delivery.
In a batch shop, every layer runs on a schedule. Collectors fire on crons. Scores update nightly. Reports assemble weekly. The arrangement works, and it has one structural flaw that no amount of better scheduling fixes: between runs, the instrument is blind. The Tuesday story surfaces in the Monday report. The rank is correct at 6am and decays all day. Every freshness problem is the same problem wearing different clothes — a batch boundary between collection and everything above it.
Apache Flink is a stream processor: a framework for running computations that never finish over events that never stop, with managed state, a serious notion of time, and delivery guarantees. It's the most mature open-source implementation of the standing-query idea this series opened with. And the precise claim I want to make is about placement: Flink slots between collection and analysis, and it replaces the schedule — nothing else.
What changes below the line
Collectors get dumber, which is a gift. Fetch, push JSON, done. All normalization, dedup, and enrichment moves into declared, reviewable queries inside the stream layer.
The questions we already ask become standing queries. Mention velocity at three time resolutions from one definition. Score deltas that announce themselves instead of waiting to be diffed. Search-position changes that surface the day they happen. Each of these currently exists as logic buried in some scheduled script; each becomes a view.
And two capabilities appear that the batch stack doesn't have at any schedule:
- Declared pattern detection. The analyst's tacit "spike that holds negative means a story is developing" becomes a written pattern, watching every brand at 3am, testable against history before it's trusted with the present. - The evidence journal. Every observation lands in an append-only log exactly once, crashes included. Any published figure becomes recomputable from the journal, which is the audit posture an instrument claiming regulatory-grade rigor actually has to back.
Recent Flink additions sweeten the placement argument. Model inference is now a SQL primitive (`CREATE MODEL` + `ML_PREDICT`), so LLM work like sentiment scoring and entity extraction happens inside the stream instead of in enrichment scripts between runs. And materialized tables (`CREATE MATERIALIZED TABLE ... FRESHNESS = INTERVAL '30' MINUTE`) invert cron logic: you declare how stale a result may be, and the system owns the refresh. A stack rank stops being "whatever last night's run produced" and becomes a table with a freshness contract.
What doesn't change above the line
This is the half of the argument that keeps a technology adoption from becoming a rewrite.
Graph analysis stays the analysis brain. Knowledge graphs answer structural questions: what the discourse around a brand is shaped like, where the gaps are, what's missing that should be present. A stream processor has no opinion about any of that. It can feed the graphs continuously (streaming entity extraction means gap analysis runs against a living graph instead of last month's corpus), and that's the entire relationship. Structure and flow are different jobs. The graph layer knows what the conversation is shaped like; the stream layer knows what just changed and how fast. An intelligence instrument needs both, and neither substitutes for the other.
The scoring methodology is untouched — streaming changes when scores update and how their movements surface, never how scoring works. The editorial layer is untouched: reports still get argued, crafted, and gated by humans, on a weekly cadence clients actually want. The report becomes the weekly reading of continuous numbers rather than the only moment the numbers exist, and the writing craft matters exactly as much as before.
The build-versus-wait question, taken seriously
A stream processor is the heaviest infrastructure decision in this whole picture: a JVM cluster, a message broker, checkpoint storage, an operational discipline. Meanwhile a cron job and a database serve a weekly cadence perfectly well. Anyone who tells you otherwise is selling something.
The threshold where continuous wins is locatable, though. It's crossed when some combination of these becomes true: clients expect Tuesday's answer on Tuesday; the count of brands times signal sources times freshness expectations no longer fits in a nightly run; a missed developing story costs more than a year of infrastructure; or the audit question ("rebuild this figure for me") starts arriving from people who won't accept "the script that computed it has since changed."
The disciplined way in is a proof phase with a falsifiable exit: run the streaming definitions beside the batch stack for one vertical, require the numbers to reconcile, and accept "wait" as a legitimate verdict. The worst adoption path is the one where the new system is declared load-bearing before it has reproduced the old system's answers.
The series, in one paragraph
Monitoring is a standing query, so the schedule is an implementation detail you can delete. Event time dates evidence by when it happened, so trends stop inheriting the crawler's schedule. The brand profile is state, so freshness stops being a question. A narrative shift is a declarable pattern, so detection goes on shift work. The weekly report is a window size, so one metric definition serves every surface reproducibly. Exactly-once is an audit feature, so every figure traces to a journal that can survive a skeptic. Put together: the observation layer becomes continuous, the judgment layers stay human, and the seam between them is exactly where a stream processor belongs.
What's still open
The threshold question is open on purpose — locating it for a real portfolio of brands, with real client expectations, is empirical work, and the proof phase is designed to answer it rather than presume it. The other open thread is organizational: continuous instruments generate continuous obligations. An alert channel that fires at 3am implies someone reads it at 3am, or an explicit decision about response latency. Infrastructure sets the tempo it's capable of; teams still have to choose the tempo they'll honor.
Related
-