Data Engineering Basics: Glossary and Tool Landscape
Try This First
Test your knowledge before reading. Don't worry if you get it wrong — that's part of learning.
Key Insights
- An introduction to data engineering: the lifecycle, modern tools, architectural patterns, and key concepts glossary.
- Covers ingestion, storage, transformation, orchestration, and data quality.
What is Data Engineering?
Data engineering is the discipline of designing, building, and maintaining systems that collect, store, transform, and make data available for analysis and machine learning. It sits at the foundation of the modern data stack, providing the infrastructure that data scientists, analysts, and business users rely on.
The Data Engineering Lifecycle
The data engineering lifecycle consists of several stages:
- Generation: Data is produced by source systems — application databases, APIs, IoT devices, event streams, log files, and third-party services.
- Ingestion: Data is extracted from sources and loaded into the data platform via batch jobs (daily/hourly exports) or streaming pipelines (real-time events via Kafka, Kinesis, or Pulsar).
- Storage: Data lands in a lakehouse (e.g., S3 + Iceberg, Delta Lake) or warehouse (Snowflake, BigQuery, Redshift) where it is organized into raw, cleaned, and aggregated layers.
- Transformation: Raw data is cleaned, normalized, joined, and reshaped using tools like dbt, Spark, or Flink. This produces analytics-ready tables and feature tables for ML.
- Serving: Clean data is exposed via APIs, dashboards (Looker, Metabase), BI tools, or feature stores (Feast, Tecton) for downstream consumers.
- Monitoring & Observability: Pipelines are tracked for freshness, volume, schema drift, data quality, and cost. Tools like Great Expectations, Soda, and Monte Carlo flag anomalies in real time.
Core Technologies in the Modern Data Stack
| Category | Tools | Purpose |
|---|---|---|
| Ingestion | Apache Kafka, Airbyte, Fivetran, Debezium | Move data from sources to the platform |
| Storage | Apache Iceberg, Delta Lake, Snowflake, BigQuery | Reliable, queryable data storage |
| Transformation | dbt, Apache Spark, SQLMesh, Flink | Clean, model, and reshape data |
| Orchestration | Airflow, Dagster, Prefect | Schedule, monitor, and manage pipeline DAGs |
| Data Quality | Great Expectations, Soda, dbt tests | Define and enforce data quality expectations |
| Observability | Monte Carlo, Datadog, OpenLineage | Monitor pipeline health and data lineage |
| Serving | Feast, Tecton, GraphQL, Metabase | Expose data to consumers |
Key Architectural Patterns
- Medallion Architecture: Organizes data into Bronze (raw ingested), Silver (cleaned/validated), and Gold (aggregated/business-ready) layers. Each layer enforces increasing data quality standards.
- Data Mesh: A decentralized architecture where domain teams own their data as products, with a shared infrastructure layer for cross-domain interoperability.
- Lambda Architecture: Combines batch and stream processing paths to balance latency and completeness. The batch layer provides accurate historical views while the speed layer handles real-time data.
- Kappa Architecture: A simplified approach where all data is treated as a stream, eliminating the separate batch layer. Stream processors handle both real-time and historical reprocessing.
Key Concepts Glossary
- Pipeline DAG
- A directed acyclic graph of tasks representing a data pipeline. Each node is a step (extract, transform, load), and edges define execution order and dependencies.
- Schema Drift
- When source data changes structure — new columns are added, types change, or fields are removed. Pipelines must handle drift gracefully to avoid breaking downstream consumers.
- Idempotency
- The property that running the same pipeline multiple times produces the same result. Essential for reprocessing and backfills without data duplication.
- Partition Pruning
- A query optimization technique where the engine skips irrelevant data partitions (e.g., by date or region) to reduce scan volume and improve query speed.
- SLA / SLO
- Service Level Agreements and Objectives define freshness, completeness, and accuracy guarantees for data products (e.g., "dashboard refreshes within 15 minutes of source update").
- Change Data Capture (CDC)
- A technique that captures row-level changes (inserts, updates, deletes) from databases in real time, enabling low-latency data synchronization.
Article Metadata
Review with Spaced Repetition
Add this lesson's 7 flashcards to your SM-2 study queue. They will appear when due in the Study Queue.
Feynman Concept Cards
Master each building block: read the ELI5, explore the analogy, work the example, find your gaps, teach it back, build it.
Data Lake is a concept in foundations. In simple terms, Data Lake covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: data lakehouse.
Analogy
Example
Find Gaps
Explain Data Lake as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Data Lake in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Data Lake should include: 1. The core components of data lake 2. How they interact 3. Expected outcomes or outputs
Batch Processing is a concept in foundations. In simple terms, Batch Processing covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: batch jo
Analogy
Example
Find Gaps
Explain Batch Processing as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Batch Processing in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Batch Processing should include: 1. The core components of batch processing 2. How they interact 3. Expected outcomes or outputs
DataOps is a concept in best practices. In simple terms, DataOps covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: DataOps practices, data operation
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Extract-Transform-Load is a concept in foundations. In simple terms, Extract-Transform-Load covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: ET
Analogy
Example
Find Gaps
Explain Extract-Transform-Load as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Transform-Load in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Transform-Load should include: 1. The core components of etl 2. How they interact 3. Expected outcomes or outputs
Extract-Load-Transform is a concept in foundations. In simple terms, Extract-Load-Transform covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: EL
Analogy
Example
Find Gaps
Explain Extract-Load-Transform as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Load-Transform in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Load-Transform should include: 1. The core components of elt 2. How they interact 3. Expected outcomes or outputs
ELT Pipeline Architecture is a concept in architecture. In simple terms, ELT Pipeline Architecture covers architectural patterns for Data Engineering. This data engineering concept addresses key topics in the architectural patterns for data engineering domain. Also known a
Analogy
Example
Find Gaps
Explain ELT Pipeline Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates ELT Pipeline Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for ELT Pipeline Architecture should include: 1. The core components of elt pipeline 2. How they interact 3. Expected outcomes or outputs
Infrastructure is a concept in specialized. In simple terms, A concept related to infrastructure
Analogy
Example
Find Gaps
Explain Infrastructure as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Infrastructure in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Infrastructure should include: 1. The core components of infrastructure 2. How they interact 3. Expected outcomes or outputs
Feynman Synthesis — Prove You Understand
1. The One-Pager
Explain this lesson's core idea to a smart 15-year-old. No jargon allowed.
2. The Gap Map
List 3 things you are still unsure about. Be specific.
Knowledge Check
Test your understanding of this lesson.
Flashcards
Space = flip · 1-4 = grade · Swipe on mobile