Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing
Key Insights
- Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.
Edit on GitHub — registry.json
Background
By 2010, LinkedIn was drowning in activity data: page views, searches, profile updates, and operational metrics needed to feed analytics, search indexing, and news feed features. Off-the-shelf messaging systems built for enterprise queues (AMQP, JMS) were too heavyweight for the volume and failed to provide the ordering guarantees consumers needed. The engineering answer, described by Kreps, Narkhede, and Rao, was to stop modeling a queue and start modeling a log.
The Design
Kafka is a distributed, partitioned, replicated commit log. Producers append records to topics; each topic is sharded into ordered partitions; consumers track their own offsets, so they can replay messages at will rather than having a broker push and delete them. Messages are not deleted when consumed — they are retained for a configured window, making the log a shared, durable buffer between systems.
Deep Dive
The paper's insight is architectural simplicity: rely on sequential disk I/O and OS page cache instead of in-memory queues or exotic storage. Appends and reads are sequential, delivering high throughput on commodity hardware; partitioning provides parallelism; replication across brokers gives durability and failover; consumer groups let a cluster of consumers share a partition load while preserving order. This yields a messaging system that treats throughput, retention, and replayability as first-class properties rather than afterthoughts.
Why It Matters
Kafka became the industry standard for event-driven data engineering precisely because it inverted the queue model: the log is the source of truth, and every consumer gets an independent view. Event sourcing, change data capture, and the "data in motion" layer of modern streaming platforms all inherit this design.
Key Takeaways
- Partition count sets the throughput ceiling and bounds per-key ordering — choose keys carefully.
- Replication factor trades durability against cost; plan for broker loss before it happens.
- Offsets make consumers replayable, which is what enables reprocessing and exactly-once patterns on top of at-least-once semantics.