Data Pipeline Patterns: Batch, Streaming, and Event-Driven Architectures

Batch Processing: The Workhorse of Data Engineering

Batch processing is the traditional approach to data processing where data is collected, stored, and processed in discrete chunks at scheduled intervals. It remains the most common pattern in data engineering due to its simplicity, reliability, and cost-effectiveness.

• When to use: Historical reporting, daily/weekly aggregations, backfills, scenarios where sub-second latency is not required.
• Common tools: Apache Spark, dbt (for SQL transformations), Airflow/Dagster/Prefect (for orchestration), Hive, Trino.
• Advantages: Simple to reason about, easy to reprocess/backfill, lower infrastructure costs, well-suited for large volumes.
• Limitations: Higher latency (minutes to hours), not suitable for real-time use cases, can be wasteful if processing only a few new records.

Batch pipelines typically follow an extract-load-transform (ELT) pattern: raw data lands in a staging area, then transformation jobs run on a schedule to clean, join, and aggregate data into analytics-ready tables.

Stream Processing: Real-Time Data in Motion

Stream processing handles data continuously as it arrives, with latencies measured in milliseconds to seconds. Every record is processed individually or in small micro-batches, enabling real-time analytics, alerting, and event-driven applications.

• When to use: Real-time dashboards, fraud detection, alerting, live personalization, monitoring and observability.
• Common tools: Apache Kafka (message broker), Apache Flink, Kafka Streams, Apache Spark Streaming, RisingWave.
• Advantages: Sub-second latency, natural fit for event-driven architectures, enables real-time decision making.
• Limitations: Higher operational complexity, state management is hard, exactly-once semantics are difficult to guarantee, typically more expensive per-record.

Stream processing requires careful handling of state (e.g., windowed aggregations), watermarks (handling late-arriving data), and checkpointing for fault tolerance.

Micro-Batch: The Middle Ground

Micro-batch processing straddles batch and streaming by processing data in small, frequent batches (e.g., every 5-60 seconds) rather than record-by-record. Apache Spark Streaming popularized this pattern with its micro-batch architecture, where each batch is a small Spark job.

• When to use: When you need near-real-time latency but want the simplicity and fault-tolerance of batch processing.
• Common tools: Spark Structured Streaming (micro-batch mode), Flink (can be configured for micro-batch).
• Trade-offs: Lower latency than traditional batch, but higher per-record overhead than true streaming due to batch job startup costs.

Lambda Architecture: Batch + Streaming

The Lambda architecture, proposed by Nathan Marz, runs batch and stream processing paths in parallel. The batch layer provides accurate, comprehensive results (eventually consistent), while the speed layer provides low-latency results. A serving layer merges outputs from both paths.

 ┌─────────────┐ ┌──────────────┐
 │ Batch Path │────→│ Batch View │──┐
 │ (Spark/SQL) │ │ (accurate) │ │ ┌──────────────┐
 └─────────────┘ └──────────────┘ ├──→│ Serving Layer│→ query
 ┌─────────────┐ ┌──────────────┐ │ │ (merged) │
 │ Stream Path │────→│ Speed View │──┘ └──────────────┘
 │ (Flink) │ │ (low-latency)│
 └─────────────┘ └──────────────┘

• Strengths: Balances accuracy and latency; the batch layer can re-process and correct speed-layer approximations.
• Weaknesses: Code complexity (maintaining two codebases), operational overhead, high infrastructure costs.
• Best for: Systems where both historical accuracy and real-time views are required — e.g., recommendation systems, fraud detection with periodic model retraining.

Kappa Architecture: Stream-Only Simplicity

The Kappa architecture, advocated by Jay Kreps (creator of Kafka), eliminates the separate batch layer. All data is treated as a stream, and the stream processor handles both real-time and historical reprocessing by replaying events from the event log.

 ┌──────────────┐ ┌──────────────┐
 │ Event Log │────→│ Stream Proc │────→ query
 │ (Kafka) │ │ (Flink/SQL) │
 └──────────────┘ └──────────────┘
 │ ↑
 └──── replay ────────┘

• Strengths: Single codebase, simpler operations, natural reprocessing by replaying the event log, lower infrastructure costs.
• Weaknesses: All state management is in-stream; complex windowed operations can be harder to implement; requires a durable, scalable event log.
• Best for: Organizations already using Kafka as their central data backbone; teams comfortable with stream processing.

Event-Driven Architecture and Change Data Capture

Event-driven architecture (EDA) is a pattern where services communicate by producing and consuming events asynchronously. In data engineering, this pattern is enabled by two key technologies: Apache Kafka (or similar message brokers) and Change Data Capture.

• Apache Kafka: A distributed event store and streaming platform. Producers write events to topics; consumers read from topics. Events are durable, ordered, and replayable. Kafka is the backbone of most modern event-driven data platforms.
• Change Data Capture (CDC): A technique that captures row-level changes (inserts, updates, deletes) from source databases in real time. Tools like Debezium sit on database transaction logs and emit change events to Kafka, enabling low-latency data synchronization without batch ETL jobs.
• Schema Registry: A companion service to Kafka that stores and enforces schemas (Avro, Protobuf, JSON Schema) for events. Producers and consumers agree on the schema, preventing data format mismatches and enabling schema evolution.

Choosing an Architecture: A Decision Framework

In practice, most modern data platforms are hybrid. They use batch for their core analytics and ML pipelines, streaming for real-time operational use cases, and micro-batch for the middle ground. The key is to start simple (batch) and add streaming where the latency requirements justify the complexity cost.