Every monitoring system I've built starts as a question someone asks repeatedly. How fast is this brand being mentioned? Did anyone new enter the top of the search results? Is the conversation about our client turning?
The standard way to automate a repeated question is a scheduled script: run the collectors, compute the answer, write it down, sleep until tomorrow. That's how most intelligence tooling works, ours included. The answer is correct at 6am and ages all day. Whatever happens between runs is invisible until the next run.
Stream processing flips the arrangement. Instead of a script that runs on a schedule, you write the question once, as a query, and the system keeps it running forever. Events arrive; the answer updates. The database world calls this a continuous query. Apache Flink, the open-source system I've been reading this week, calls the whole discipline stream processing, and it's the most mature implementation of the idea I've found.
Here's the version of the shift that made it click for me. A weekly mention count is this, in spirit:
```sql SELECT brand, COUNT(*) FROM mentions WHERE week = last_week; ```
run by a cron job every Monday. The streaming version is the same question with the schedule deleted:
```sql SELECT brand, window_start, COUNT(*) FROM TABLE(TUMBLE(TABLE mentions, DESCRIPTOR(event_time), INTERVAL '7' DAY)) GROUP BY brand, window_start, window_end; ```
That query never finishes. It emits a row whenever a week's worth of evidence is complete, and it will emit next week's row without anyone re-running anything. The question became infrastructure.
The practical difference shows up in what you stop maintaining. A scheduled pipeline needs orchestration: did last night's run finish, did it half-fail, is the number on the dashboard from the good run or the broken one? A standing query has none of those states. It's either running and current, or down and loudly so.
The deeper difference is what it does to your relationship with the question. When a question costs a script and a cron slot, you ration questions. When a question is a view definition, you ask more of them, at more time resolutions, and you keep the good ones running permanently. The interesting metrics in intelligence work are almost all rates and changes, and rates and changes are exactly what standing queries are good at.
The rest of this series takes one streaming concept per post and grounds it in a concrete monitoring problem: event time (why late-discovered evidence backdates correctly), state (the brand profile as a living accumulator), patterns (a narrative shift as a detectable shape), windows (the weekly report as one window size among many), and exactly-once processing (why an audit trail wants a stream processor underneath it).
What's still open
The honest question about all of this is operational: a stream processor is heavy infrastructure, and a cron job plus a database serves a weekly cadence perfectly well. The threshold where continuous wins is a real design question, and I'd rather locate it precisely than assume the answer. That thread runs through the whole series.
Related
-