Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Key Insights
- Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.
Edit on GitHub — registry.json
Background
By 2012, MapReduce had made petabyte-scale processing routine, but its disk-centric execution model punished the workloads that mattered next: iterative algorithms and interactive exploration. Machine learning, graph analysis, and iterative data mining repeatedly reuse intermediate results across stages, and each pass forced a write to stable storage. Zaharia and the Berkeley AMPLab designed Spark around a single question: what if the fault-tolerant abstraction itself lived in memory?
The RDD Abstraction
The paper introduces Resilient Distributed Datasets (RDDs): immutable, partitioned collections of records that support coarse-grained transformations such as map, filter, and join, plus actions like count and collect. Instead of replicating data for fault tolerance, an RDD records its own lineage — the dependency graph of transformations that produced it — so any lost partition can be recomputed from its parents. Lineage replaces replication as the recovery mechanism, which is what makes in-memory caching practical.
Deep Dive
Users can mark RDDs for persistence with persist(), choosing between memory-only, memory-and-disk, or serialized layouts. Lazy evaluation defers computation until an action is invoked, letting the scheduler pipeline transformations and recover only the partitions actually lost. The paper reports 10-100x speedups over Hadoop on iterative workloads such as logistic regression and k-means clustering, with recovery an order of magnitude faster than replication-based approaches.
Why It Matters
RDDs were the fault-tolerance core that later became Spark SQL, DataFrames, and Structured Streaming. The lineage-based recovery idea now underpins every modern lakehouse engine and structured streaming system, and remains the mental model for debugging recomputation and caching behavior in production Spark clusters.
Key Takeaways
- Lineage beats replication for recomputable transformations — recovery cost trades off against memory pressure.
- Cache deliberately: memory-only persistence avoids disk I/O but drops partitions under pressure; plan for spill.
- Coarse-grained transformations are the price of lineage: RDDs support batch updates well, point updates poorly.