Data Engineering Glossary

Core Concepts

AI Conformity Assessment & Auditing

AI Conformity Assessment & Auditing Also: AI auditing, conformity assessment, AI compliance audit, algorithm audit

Acid Transaction Management

A concept related to acid-transaction-management Also: acidtransactionmanagement

Apache Arrow / Parquet

Apache Arrow / Parquet Also: Arrow, Parquet, columnar storage

Apache Flink

Apache Flink Also: Flink

Apache Iceberg

Apache Iceberg Also: Iceberg, table format

Apache Kafka

Apache Kafka Also: Kafka

Batch & Stream Merging

Batch & Stream Merging Also: lambda architecture, kappa architecture, unified batch streaming

Batch Processing

Batch Processing Also: batch jobs, scheduled processing

Change Data Capture

Change Data Capture Also: CDC, change-data-capture

Change Data Capture Patterns

Change Data Capture Patterns Also: CDC patterns, debezium, log-based CDC, incremental

DORA — ICT Risk Management

DORA — ICT Risk Management Also: DORA, Digital Operational Resilience Act, ICT risk, operational resilience

Dagster Orchestrator

Dagster Orchestrator Also: dagster

Data Catalog

Data Catalog Also: data cataloging, metadata catalog, data discovery, data inventory

Data Contract Testing

Data Contract Testing Also: contract testing, schema testing, data quality gates

Data Contracts

Data Contracts Also: data contract, SLA for data

Data Cost Intelligence

Data Cost Intelligence Also: FinOps, data cost optimization, cost attribution

Data Discovery & Cataloging

Data Discovery & Cataloging Also: data catalog, metadata management, data marketplace

Data Governance

Data Governance Also: data catalog, metadata management, data lineage

Data Lake

Data Lake Also: data lakehouse

Data Lineage

Data Lineage Also: data provenance, data tracing, column lineage, impact analysis

Data Mesh

Data Mesh Also: data mesh architecture

Data Mesh Governance

Data Mesh Governance Also: federated governance, domain ownership, data product governance

Data Observability

Data Observability Also: data monitoring, data health, observability

Data Pipeline Architecture

Data Pipeline Architecture Also: pipeline architecture, data pipeline design

Data Platform KPIs

Data Platform KPIs Also: data maturity, platform metrics, data reliability

Data Quality

Data Quality Also: data observability, data validation

Data Quality Frameworks

Data quality frameworks, pioneered by Wang and Strong (1996), provide a systematic taxonomy of data quality dimensions including intrinsic (accuracy, objectivity, believability), contextual (relevancy, timeliness, completeness), representational (interpretability, consistency), and accessibility (access, security) dimensions. Also: Wang-Strong 1996, data quality dimensions, DQ framework, beyond accuracy, data-quality-framework

Data Quality Monitoring

Data Quality Monitoring Also: data observability, DQ dashboards, data SLAs

Data Quality SLAs

Data Quality SLAs Also: data SLA, data reliability, data uptime

Data Security & Access Control

Data Security & Access Control Also: RBAC, data encryption, access control, data masking

Data Versioning

Data Versioning Also: data version control, dataset versioning, data lineage version

Data Warehouse

Data Warehouse Also: DWH, analytical data store

Data Warehouse Design Patterns

Data Warehouse Design Patterns Also: warehouse design, dimensional modeling, star schema

DataOps

DataOps Also: DataOps practices, data operations, data lifecycle

Debiasing Pipeline

Debiasing Pipeline Also: bias mitigation, fairness pipeline, bias detection, algorithmic fairness

Differential Privacy

Differential Privacy Also: DP, epsilon-delta privacy, privacy budget, noise injection

Distributed Systems for Data

Distributed Systems for Data Also: distributed computing, consensus, sharding

ELT Pipeline Architecture

ELT Pipeline Architecture Also: ELT, load then transform, modern ELT

EU AI Act — High-Risk Classification

EU AI Act — High-Risk Classification Also: AI Act, high-risk AI, EU AI Act, AI risk classification

EU Data Act — Data Interoperability

EU Data Act — Data Interoperability Also: EU Data Act, data interoperability, data portability, smart contract safeguards

Enterprise Architecture

A concept related to enterprise-architecture Also: enterprisearchitecture

Extract-Load-Transform

Extract-Load-Transform Also: ELT

Extract-Transform-Load

Extract-Transform-Load Also: ETL, extract transform load

Fairness Metrics in ML

Fairness Metrics in ML Also: algorithmic fairness, demographic parity, equal opportunity, fairness metric

Feature Store

Feature Store Also: ML feature store, feature engineering, feature serving

Federated Learning

Federated Learning Also: federated ML, distributed training, privacy-preserving ML, federated averaging

GDPR Anonymization & Pseudonymization

GDPR Anonymization & Pseudonymization Also: anonymization, pseudonymization, data masking, de-identification

Infrastructure

A concept related to infrastructure Also: infrastructure

Lakehouse Architecture

Lakehouse Architecture Also: data lakehouse, lakehouse paradigm, unified analytics

Lakehouse Architecture

The Lakehouse architecture, formalized by Armbrust et al. (2021), combines the flexibility of data lakes (cheap object storage, diverse data types) with the reliability of data warehouses (ACID transactions, schema enforcement, performance optimization). It achieves this through a new open table format layer (Delta Lake, Iceberg, Hudi) that provides ACID transactions on cloud storage. Also: Armbrust 2021, lakehouse, delta lakehouse, open lakehouse, lakehouse architecture, lakehouse-architecture

Lakestream Architecture

Lakestream Architecture represents the convergence of streaming and lakehouse paradigms, where a single copy of data in Kafka topics simultaneously serves real-time stream processing and Iceberg table reads. Formalized by StreamNative in 2026, it achieves 95% cost reduction by eliminating separate streaming and batch storage layers. The default 2026 stack is Debezium (CDC) → Kafka (transport) → Flink (processing) → Iceberg (storage). Extends the lakehouse architecture pioneered by Armbrust et al. (2021) with first-class streaming semantics. Also: lakestream, StreamNative Lakestream, streaming-first lakehouse, streaming-lakehouse, stream batch unification

ML Pipeline Engineering

ML Pipeline Engineering Also: MLOps, feature engineering, model deployment

MapReduce Programming Model

MapReduce, introduced by Dean and Ghemawat (2004), is a programming model for processing large datasets in parallel across distributed clusters. It abstracts distributed computation into two phases — map (filter/transform) and reduce (aggregate/summarize) — hiding complexities of parallelization, fault tolerance, and data distribution from the programmer. Also: Dean-Ghemawat 2004, MapReduce, map reduce, google mapreduce, map-reduce

Metric Store

Metric Store Also: business metrics, metric platform, KPIs metadata

Model Cards & Documentation Standards

Model Cards & Documentation Standards Also: model card, model documentation, model transparency report, model governance

NIS2 Directive — Cybersecurity Resilience

NIS2 Directive — Cybersecurity Resilience Also: NIS2, NIS 2 Directive, cyber resilience, network security

Pipeline Cost Optimization

Pipeline Cost Optimization Also: cost optimization, data FinOps, pipeline efficiency

Query Optimization

Query Optimization Also: SQL optimization, query tuning, execution planning

Real-Time Analytics

Real-Time Analytics Also: real-time reporting, live dashboards, streaming analytics

Real-Time Analytics Architecture

Real-Time Analytics Architecture Also: real-time OLAP, streaming analytics, druid, clickhouse

Record Linkage & Entity Matching

Record linkage, formalized by Fellegi and Sunter (1969), is the methodology for identifying records that refer to the same entity across different data sources. It uses probabilistic matching rules based on the agreement patterns of common fields, providing the mathematical foundation for modern entity resolution and deduplication. Also: Fellegi-Sunter, entity matching, data matching, record linkage, probabilistic linkage

Reverse ETL

Reverse ETL Also: operational analytics, data activation, warehouse to SaaS

Schema Migration Strategies

Schema Migration Strategies Also: schema evolution, backward compatibility, zero-downtime migration

Schema Registry

Schema Registry Also: schema evolution, schema management

Stream Processing

Stream Processing Also: real-time processing, stream processing

Stream Processing & The Dataflow Model

The Dataflow Model, introduced by Akidau et al. (2015), provides a unified framework for processing both batch and streaming data. It defines a general approach to handling unbounded, out-of-order data by separating the what (computation via operators), where (windowing), when (triggering), and how (accumulation) of stream processing. Also: Dataflow model, Akidau 2015, Beam model, unbounded processing, stream-processing-model, Apache Beam, stream processing theory

Synthetic Data Generation for GDPR Compliance

Synthetic Data Generation for GDPR Compliance Also: synthetic data, data generation, generative modeling, synth data

Ux Design Patterns

A concept related to ux-design-patterns Also: uxdesignpatterns

Workflow Orchestration

Workflow Orchestration Also: workflow automation, pipeline orchestration, DAG

dbt (data build tool)

dbt (data build tool) Also: data build tool

Relationships

• AI Conformity Assessment & Auditing: implements ML Pipeline Engineering, requires Model Cards & Documentation Standards, requires ML Pipeline Engineering
• Apache Arrow / Parquet: enables Data Lake, requires Lakehouse Architecture
• Apache Flink: implements Stream Processing, requires Real-Time Analytics
• Apache Iceberg: implements Data Lake, requires Lakehouse Architecture
• Apache Kafka: enables Stream Processing, requires Stream Processing
• Batch & Stream Merging: requires Stream Processing, related_to Batch Processing
• Batch Processing: requires Data Pipeline Architecture
• Change Data Capture: related_to Batch Processing, requires Apache Kafka
• Change Data Capture Patterns: implements Change Data Capture, requires Change Data Capture
• DORA — ICT Risk Management: requires Data Governance
• Dagster Orchestrator: implements Extract-Transform-Load, implements Extract-Load-Transform, requires DataOps
• Data Catalog: implements Data Governance, enables Data Lineage, requires Data Mesh
• Data Contract Testing: implements Data Contracts, requires Data Contracts
• Data Contracts: enables Data Quality, requires Data Governance
• Data Cost Intelligence: implements DataOps, requires Pipeline Cost Optimization
• Data Discovery & Cataloging: implements Data Catalog, requires Data Catalog
• Data Governance: requires Data Mesh, requires Data Quality
• Data Lake: requires Data Warehouse
• Data Lineage: implements Data Governance, requires Data Observability
• Data Mesh: requires Data Contracts, requires Lakehouse Architecture
• Data Mesh Governance: implements Data Mesh, requires Data Mesh
• Data Observability: implements Data Quality, requires Data Pipeline Architecture
• Data Pipeline Architecture: implements Extract-Transform-Load, related_to Stream Processing
• Data Platform KPIs: requires Data Observability, requires Data Warehouse
• Data Quality: requires Data Pipeline Architecture
• Data Quality Frameworks: part_of Data Quality
• Data Quality Monitoring: implements Data Quality, requires Data Quality
• Data Quality SLAs: implements Data Quality, requires Data Quality Monitoring
• Data Security & Access Control: part_of Data Governance, requires Data Governance
• Data Versioning: enables Data Lineage, requires Data Lake
• Data Warehouse: requires Data Pipeline Architecture
• Data Warehouse Design Patterns: implements Data Warehouse, requires Data Warehouse
• DataOps: enables Data Pipeline Architecture, requires Data Observability, requires Data Pipeline Architecture
• Debiasing Pipeline: requires Fairness Metrics in ML, implements ML Pipeline Engineering, requires Data Quality
• Differential Privacy: implements GDPR Anonymization & Pseudonymization, requires Data Security & Access Control
• Distributed Systems for Data: requires Data Mesh, enables Stream Processing, requires Data Pipeline Architecture
• ELT Pipeline Architecture: implements Extract-Load-Transform, requires Extract-Load-Transform
• EU AI Act — High-Risk Classification: requires AI Conformity Assessment & Auditing
• EU Data Act — Data Interoperability: requires Data Contracts, requires Schema Registry, requires Data Mesh
• Extract-Load-Transform: supersedes Extract-Transform-Load, requires Data Pipeline Architecture
• Extract-Transform-Load: requires Extract-Load-Transform
• Fairness Metrics in ML: requires Debiasing Pipeline
• Feature Store: requires ML Pipeline Engineering, requires Data Catalog
• Federated Learning: related_to Differential Privacy, requires Differential Privacy
• GDPR Anonymization & Pseudonymization: enables Synthetic Data Generation for GDPR Compliance, implements Data Security & Access Control, requires Data Governance
• Lakehouse Architecture: supersedes Data Lake, implements Apache Iceberg, requires Data Lake, enables Apache Iceberg, part_of Lakehouse Architecture
• Lakestream Architecture: requires Apache Flink, requires Apache Iceberg, requires Apache Kafka, supersedes Lakehouse Architecture, enables Stream Processing & The Dataflow Model
• ML Pipeline Engineering: related_to Data Pipeline Architecture, requires Workflow Orchestration, requires Data Pipeline Architecture
• MapReduce Programming Model: enables Data Pipeline Architecture
• Metric Store: implements Data Observability, requires Data Warehouse
• Model Cards & Documentation Standards: implements ML Pipeline Engineering, requires Debiasing Pipeline
• NIS2 Directive — Cybersecurity Resilience: requires DORA — ICT Risk Management
• Pipeline Cost Optimization: implements Data Cost Intelligence, requires Data Platform KPIs
• Query Optimization: enables Data Warehouse, requires Data Warehouse Design Patterns
• Real-Time Analytics: enables Stream Processing, requires Stream Processing
• Real-Time Analytics Architecture: implements Real-Time Analytics, requires Real-Time Analytics
• Record Linkage & Entity Matching: enables Data Quality, enables entity-resolution
• Reverse ETL: enables Data Warehouse, requires Data Warehouse
• Schema Migration Strategies: enables Schema Registry, requires Data Versioning
• Schema Registry: implements Data Contracts, requires Schema Migration Strategies
• Stream Processing: related_to Batch Processing, requires Data Pipeline Architecture
• Stream Processing & The Dataflow Model: part_of Stream Processing
• Synthetic Data Generation for GDPR Compliance: requires Data Quality, enables Debiasing Pipeline, requires GDPR Anonymization & Pseudonymization
• Workflow Orchestration: related_to Dagster Orchestrator, enables Data Pipeline Architecture, requires Data Pipeline Architecture
• dbt (data build tool): implements Extract-Load-Transform, requires DataOps

Authoritative Sources

• Apache Flink — Flink documentation and release notes (https://flink.apache.org)
• Apache Iceberg — Iceberg table format — specs, REST catalog, performance (https://iceberg.apache.org)
• Apache Kafka — Kafka documentation, KIPs, and ecosystem updates (https://kafka.apache.org)
• ApplyData — Data & AI engineering trends — multimodal lakehouses, EDD, context engineering (https://applydata.io/blog)
• AWS Big Data Blog — AWS analytics services — Glue, EMR, Kinesis, Athena (https://aws.amazon.com/blogs/big-data/)
• ClickHouse Blog — ClickHouse — columnar OLAP, vector search, performance engineering (https://clickhouse.com/blog)
• Confluent Blog — Kafka ecosystem — Schema Registry, Flink SQL, data streaming (https://www.confluent.io/blog)
• Dagster Blog — Dagster orchestration — software-defined assets, IO managers (https://dagster.io/blog)
• Data Engineer Things — Practitioner newsletter — data engineering trends, tooling, community (https://dataengineerthings.substack.com)
• Databricks Blog — Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog (https://www.databricks.com/blog)
• Dataverses Blog — Data engineering trends reports, lakehouse benchmarks, AI-native platforms (https://dataverses.io/resources/blog)
• dbt Blog — dbt Labs engineering blog — analytics engineering, Semantic Layer (https://www.getdbt.com/blog)
• DuckDB — DuckDB — in-process analytics, extensions, blog on query optimization (https://duckdb.org)
• GCP Data Blog — Google Cloud data analytics — BigQuery, Dataflow, Dataproc (https://cloud.google.com/blog/products/data-analytics)
• Iceberg Lakehouse Blog — Apache Iceberg deep dives — v4 development, ecosystem, community (https://iceberglakehouse.com)
• LakeOps Blog — Open Apache Lakehouse — modular architecture, vendor-neutral platforms (https://lakeops.dev/blog)
• Meltano Blog — Open-source ELT — Singer taps, Meltano SDK, data integration (https://meltano.com/blog)
• Snowflake Blog — Snowflake — data warehousing, Snowpipe, dynamic tables (https://www.snowflake.com/blog)
• StreamNative Blog — Lakehouse-native streaming, Ursa for Kafka, Lakestream architecture (https://streamnative.io/blog)
• Trino — Distributed SQL query engine — releases, query federation, performance (https://trino.io)