Data Engineering & Infrastructure
Data pipelines, orchestration, quality engineering, streaming, storage, and analytics infrastructure.
Not sure where to start? Take the 2-minute diagnostic to be placed into a learning mode tuned to your level.
Take the quiz →📖 Key Terms (4)
🧠 Concepts (15)
📚 Learning Paths (6)
PySpark Fundamentals: Distributed Data Processing at Scale
Master PySpark DataFrame API, Spark SQL, and distributed processing patterns for building production
Partitioning Strategies: Optimizing Data Layout for Performance
Learn how partitioning strategies affect query performance, storage efficiency, and cost in modern d
Building Resilient Data Pipelines with Spark and dbt
Combine Spark's processing power with dbt's transformation layer to build production-grade, tested,
Data Contracts & Data Lineage: Building Trust in Modern Data Pipelines
Data contracts formalize the agreement between data producers and consumers, while data lineage trac
Data Mesh: Architecture, Governance, and Real-World Adoption
Data mesh applies domain-driven design to data architecture, organizing ownership by business domain
Change Data Capture: Real-Time Sync Patterns and Tools
Change Data Capture (CDC) captures row-level changes in databases and streams them to downstream sys
All Articles
Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout
Head-to-head comparison of the three major lakehouse storage formats: table mutation semantics, time travel, schema evolution, compaction, and ecosystem integration (Spark, Flink, Trino, DuckDB). Benchmark results included.
The development pipeline is a production system
An argument for operating CI/CD pipelines with production discipline: observability, SLOs, capacity planning, and incident response for the build-and-release path. Treats the development pipeline as a first-class system whose downtime directly blocks engineering throughput.
Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free
Release notes for Rune 1.1, a developer tool that now embeds Python support, ships an Emacs-compatible editor mode, adds a symbol index for fast code navigation, and moves to a free pricing model. Useful signal for tooling teams evaluating in-process editing and analysis environments.
CosmosEscape: Taking over Every Database in Azure Cosmos DB
A security research disclosure (CosmosEscape) demonstrating sandbox escape in Azure Cosmos DB: chaining flaws in the JavaScript user-defined function / stored procedure execution environment to move from one database tenant to others. Highlights the risk surface of multi-tenant managed databases that expose scriptable execution.
Show HN: A local merge queue for parallel Claude Code agents
A Show HN for a local merge queue that coordinates parallel Claude Code agents. Multiple AI agents editing the same repository need serialized integration to avoid conflicting writes; a local merge queue stages, merges, and replays agent outputs so parallelism does not corrupt the working tree. Relevant to teams running concurrent AI coding agents.
Choose DuckDB rather than SQLite
A comparison argument for choosing DuckDB over SQLite. Where SQLite is a row-oriented OLTP engine optimized for point lookups and transactional writes, DuckDB is a columnar, vectorized OLAP engine designed for analytical queries over large datasets. Includes the practical implications for aggregate-heavy workloads, Parquet reads, and in-process analytics.
SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers
A production-focused guide to running SQLite for low-latency application servers. Covers WAL (write-ahead logging) mode for concurrent readers and a single writer, busy_timeout and checkpoint tuning to keep WAL files bounded, and custom VFS layers that intercept I/O for durability, encryption, or storage-tier routing. Practical reference for teams that push SQLite beyond its embedded default configuration.
Debezium and CDC: Capturing Database Changes at Scale
Change Data Capture with Debezium: connector configuration, schema evolution handling, initial snapshots, and integration with Kafka and Flink. Patterns for reliable replication and exactly-once semantics.
The Lakestream Paradigm: How Streaming-First Lakehouse Architecture Is Replacing Batch ETL in 2026
By July 2026, the streaming-first lakehouse (Lakestream) has become the dominant data architecture pattern. The default stack — Debezium for CDC, Kafka for transport, Flink for processing, Iceberg for storage — unifies real-time and batch workloads on a single copy of data. StreamNative's architecture achieves 95% cost reduction by eliminating separate streaming and batch storage. RisingWave collapses CDC, processing, and serving into one system. Flink CDC 3.6.0 delivers enhanced schema evolution for AI pipelines. Multimodal lakehouses (LanceDB) extend the paradigm to vectors, video, and audio.
Fellegi & Sunter (1969) - A Theory for Record Linkage
Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.
Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers
Wang and Strong develop the first comprehensive data quality framework, identifying 15 dimensions in four categories (intrinsic, contextual, representational, accessibility), changing how organizations think about data quality beyond accuracy.
Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters
MapReduce abstracts distributed computation into map and reduce phases, enabling petabyte-scale data processing across commodity clusters while hiding parallelization, fault tolerance, and load balancing complexities.
Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data
Bigtable is a distributed storage system for petabyte-scale structured data across thousands of commodity servers, providing a sparse, distributed, persistent multi-dimensional sorted map that influenced the NoSQL revolution.
Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost
The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.
Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms
The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.
Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing
Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.
Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.
Apache Iceberg - Open Table Format Specification
Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.
Stonebraker et al (2018) - What Goes Around Comes Around: Data Management Cycles
A historical analysis of data management systems tracing the cycles from hierarchical to relational to NoSQL to NewSQL, arguing that specialization (one size does not fit all) is the enduring lesson.
Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era
How Apache Iceberg enables ACID transactions on data lakes: partitioning, hidden partitioning, time travel, snapshot isolation, and Iceberg REST catalog. Migration strategies from Hive-style tables and Parquet-only storage.
Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture
Production patterns for Apache Kafka: topic design strategies, consumer group rebalancing, exactly-once semantics, Kafka Connect for source/sink integration, and ksqlDB for stream processing. Real case studies from financial services.
Data Contracts: Schema as API for the Analytics Team
Implementing data contracts with dbt-expectations, Soda, and Great Expectations. Covers schema evolution, contract versioning, producer/consumer ownership patterns, and breaking change detection in production pipelines.
Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026
Comprehensive comparison of the three leading Python orchestrators: execution model, DAG vs asset paradigm, scaling characteristics, monitoring, and community ecosystem. Decision framework for greenfield and migration scenarios.
Data Quality at Scale: Great Expectations Beyond Unit Tests for Data
Advanced patterns for Great Expectations in production: custom expectations, data docs auto-generation, checkpoint orchestration, and integration with Dagster and Airflow for automated quality gating.
2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch
Predictions for data engineering in 2027: AI-assisted pipeline generation, universal catalogs with Unity Catalog and Iceberg REST, real-time streaming replacing nightly batches, and the convergence of data and ML platforms.
Data Platform as a Product: UX Patterns for Internal Developer Platforms
Treating the data platform as an internal product: developer experience design, self-service data ingestion, catalog/search UX, pipeline debugging tools, and SLA dashboards. Patterns from leading platform teams.
The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack
The analytics engineering discipline: how dbt and SQLMesh transformed the data workflow, the shift from ETL to ELT, analytics engineering best practices, and the evolving role between data engineering and data science.
Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale
Strategies for reducing data pipeline costs: intelligent partitioning, incremental processing, compute auto-scaling, storage tiering (Iceberg maintenance), query optimization, and workload scheduling on spot/preemptible instances.
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production Schema registry architectures enable versioned, contract-enforced data serialization across distributed pipelines. This document provides a comprehensive reference for implementing schema evolution with backward, forward, and full compatibility contracts in production data engineering environments. Overview Schema registry architectures with Confluent Schema Registry and Apicurio provide centralized schema management
Dagster 2.0: Next-Gen Data Pipeline Orchestration for the Modern Data Platform
Deep dive into Dagster 2.0's asset-based orchestration model, software-defined assets, and the shift from DAG-centric to asset-centric pipeline design. Covers partitioning, backfills, and the new dagster-ui 2.0.
Data Products: Designing APIs for the Internal Data Platform
Data product design patterns: API contracts, SLAs, versioning, discovery, and access control. Implementation with dbt (data products as models), Dagster (software-defined assets), and DataHub for cataloging.
Data Mesh in Practice: Implementing Domain Ownership Without Chaos
Practical guide to data mesh adoption: domain ownership patterns, data product definitions, federated governance, and the compute platform. Case studies of mesh implementations and common failure modes.
Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform
Infrastructure as Code patterns for data platforms: Terraform modules for Kafka clusters, Iceberg catalogs, dbt Cloud projects, and Dagster deployments. State management, CI/CD for infrastructure, and multi-environment strategies.
Kubernetes for Data Engineering: Running Data Pipelines on K8s
Running data workloads on Kubernetes: Airflow Executor types (Celery vs Kubernetes), Dagster on K8s, Spark on Kubernetes with the Spark Operator, and stateful workloads (Kafka, Flink) on K8s. Resource management and cost optimization.
Feature Stores at Scale: Feast vs Tecton in Production Deployments
Deep comparison of Feast (open source) and Tecton (managed): feature definitions, online/offline serving, point-in-time correctness, stream feature computation, and cost models. Production deployment patterns.
ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow
Production ML pipeline patterns: Feast feature serving for training/inference consistency, MLflow model registry and deployment, and Dagster for ML pipeline orchestration. Feature engineering at scale with dbt and Spark.
Data Observability: Monitoring, Lineage, and Incident Response for Pipelines
Implementing data observability with open source tools: OpenLineage for lineage, Great Expectations for quality monitoring, and custom health checks. Incident response runbooks and SLAs for data products.
Building a Data Platform on a Budget: The Open Source Stack in 2026
Complete open source data stack: Dagster + dbt + Iceberg + Trino + DuckDB + Superset. Cost analysis against Snowflake and Databricks. Deployment patterns with Docker Compose, Terraform, and Kubernetes.
SQLMesh: The SQL-First Data Transformation Framework Challenging dbt
SQLMesh's approach to data transformation: physical vs logical plans, virtual data environments, automatic column-level lineage, and backward-incompatible change detection. Comparison with dbt's materialization model.
dbt Mesh: Decentralizing Data Transformation at Enterprise Scale
dbt Mesh architecture: dbt projects as domains, cross-project refs, blue/green deployments for models, and governance through the dbt Cloud Discovery API. Migration guide from monolithic dbt projects.
Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster
Applying modern DataOps orchestration to genomics: Dagster assets for sequencing pipeline stages, Great Expectations for quality gates on base-call accuracy, and dbt for cohort-level analytical transformations.
Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery
Applying DataOps and MLOps principles to computational biology workflows: version-controlled training data, reproducible feature engineering, automated model validation, and pipeline CI/CD for CRISPR target prediction.
EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01
Trending (HackerNews, 2026-06-01) EY Canada published a cybersecurity report and most citations were hallucinated (discussion) (320 pts) White House's Aliens.gov Site Brags That ICE Arrested More