Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost
Key Insights
- The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.
Edit on GitHub — registry.json
Background
Stream processing used to force a choice: process data with low latency and accept approximate results, or process it exactly and accept batch delays. Akidau and colleagues — the team behind Google Cloud Dataflow and later Apache Beam — set out to show the choice was an implementation artifact, not an inherent law. Their answer is the Dataflow model: one framework for batch and stream, built on four questions every pipeline must answer.
The Four Dimensions
What: the data being computed, including how it is windowed. Where: the point in event time where computation happens. When: the point in processing time when results are emitted. How: how refinements of the results relate — discarded, accumulated, or accumulated-and-retracted. Separating these dimensions is the model's core move.
Deep Dive
The model's practical payload is watermarks: estimates of event-time completeness for unbounded, out-of-order data. Watermarks allow a pipeline to emit correct results on schedule, then correct them as late data arrives within a configurable allowed-lateness window. Trigger semantics control emission timing — after window completion, periodically, or on data arrival — and accumulation modes define how partial results update. With these mechanisms, the same pipeline code expresses batch (bounded input) and stream (unbounded input) uniformly, and correctness, latency, and cost become explicit tradeoffs the user sets rather than hidden engine decisions.
Why It Matters
Apache Beam brought this model to open source, and every serious streaming system since — Flink, Spark Structured Streaming, Kafka Streams — incorporates pieces of it, particularly event-time processing and watermarking. It remains the reference for reasoning about out-of-order data.
Key Takeaways
- Window in event time, trigger in processing time — never conflate the two clocks.
- Watermarks are heuristics: set allowed lateness with real downstream tolerance in mind.
- Accumulation modes decide whether updates replace or append — choose before designing sinks.