A mention gets published Friday night. Our collector finds it Monday morning. Which day's number does it belong to?
Every batch monitoring system I've seen answers "Monday," silently, because batch systems count what the run found, and the run happened Monday. Multiply that by every slow-to-surface piece of evidence (forum posts, reviews, transcripts, anything a crawler reaches late) and your trend lines are systematically smeared rightward. A quiet Friday followed by a loud Monday might just be your crawler's schedule.
This has a name in stream processing, and a clean solution. The distinction is event time versus processing time: when the thing happened versus when your system learned about it. Flink lets you declare, per stream, which timestamp governs the math, and then handles the consequences.
The consequences are the interesting part. If Friday's mention can arrive on Monday, then Friday's count isn't final until you've waited long enough for stragglers. How long is long enough? That's a declared policy called a watermark, and I find it clarifying precisely because it forces the question into the open:
```sql WATERMARK FOR published_at AS published_at - INTERVAL '3' DAY ```
That line says: evidence for a given day keeps being accepted for three days; after that, the day's numbers close. Both halves are a policy you chose, wrote down, and can defend. Compare that to the batch status quo, where the effective policy ("things count for whenever we happened to find them") is nowhere in the code and nobody chose it.
There's a genuine tension here, and the right move is to run both clocks for different jobs. Analysis wants event time: a week-over-week mention trend should reflect when the conversation actually happened, even if the numbers take three days to settle. Alerting wants discovery time: if a damaging thread from last Tuesday surfaces today, the alert should fire today. Same stream, two timestamps, both first-class.
The intelligence-work translation: event time is an evidence-dating discipline. A report that says "mentions rose in the week of 2026-06-22" is making a claim about the world, and the claim should be anchored to when things happened in the world. Attribution to the collector's schedule is a subtle form of being wrong that nobody catches, because the totals still add up.
What's still open
Three days is a guess. The principled version measures the actual discovery-lag distribution per source and sets the watermark from data. That measurement is itself a nice standing query.
Related
-