DataOps Glossary
Key Insights
- Key DataOps and data engineering terminology.
Key terms used across DataOps, data engineering, and data science — as referenced in AcaciaFund's research and learning materials.
- DataOps
- Automated, process-oriented methodology applying Agile, DevOps, and statistical process control to data pipeline lifecycle management.
- ELT vs ETL
- Extract-Load-Transform: data is extracted and loaded raw into the warehouse, then transformed in-place. Contrasts with classic ETL where transformation happens before loading.
- Medallion Architecture
- Bronze (raw) → Silver (cleaned) → Gold (aggregated) data layering pattern popularized by Databricks for lakehouse implementations.
- Pipeline Observability
- Real-time monitoring of data pipelines — row counts, schema drift, latency, error rates — to detect and diagnose failures quickly.
- Data Contract
- Formal agreement between data producers and consumers specifying schema, semantics, quality SLOs, and ownership.
- dbt
- Data build tool — SQL-first transformation framework. Models are SELECT statements; dbt handles materialization, testing, and docs.
- Great Expectations
- Open-source Python framework for defining, testing, and documenting data quality expectations.
- SQI (Signal Quality Index)
- AcaciaFund's composite metric: source authority × freshness × consensus × relevance, normalized to [0,1].
- CDC (Change Data Capture)
- Technique for capturing row-level changes in databases, enabling real-time data replication without bulk loads.
- DAG (Directed Acyclic Graph)
- Acyclic graph of tasks with dependencies — the fundamental scheduling unit in Airflow, Dagster, and Prefect.
- MCP (Model Context Protocol)
- Emerging protocol for AI agents to interact with data tools and APIs in a standardized way, enabling agent-operated data pipelines.
- Feature Store
- Centralized repository for ML features enabling consistent computation, sharing, and serving across training and inference.
- Data Lineage
- End-to-end tracking of data from source through transformations to final consumption, enabling debugging, auditing, and impact analysis.
- Schema-on-Read
- Data lake paradigm where schema is applied at query time rather than ingest time, enabling flexible data storage.
Article Metadata
Further Reading
Databricks Blog
Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog
Apache Kafka
Kafka documentation, KIPs, and ecosystem updates
Apache Flink
Flink documentation and release notes
Apache Iceberg
Iceberg table format — specs, REST catalog, performance
dbt Blog
dbt Labs engineering blog — analytics engineering, Semantic Layer
Dagster Blog
Dagster orchestration — software-defined assets, IO managers
Feynman Concept Cards
Master each concept: read the ELI5, explore analogies, work examples, and teach it back.
Extract-Load-Transform is a concept in foundations. In simple terms, Extract-Load-Transform covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: EL
Analogy
Example
Find Gaps
Explain Extract-Load-Transform as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Load-Transform in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Load-Transform should include: 1. The core components of elt 2. How they interact 3. Expected outcomes or outputs
DataOps is a concept in best practices. In simple terms, DataOps covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: DataOps practices, data operation
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Extract-Transform-Load is a concept in foundations. In simple terms, Extract-Transform-Load covers foundational knowledge in Data Engineering. This data engineering concept addresses key topics in the foundational knowledge in data engineering domain. Also known as: ET
Analogy
Example
Find Gaps
Explain Extract-Transform-Load as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Transform-Load in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Transform-Load should include: 1. The core components of etl 2. How they interact 3. Expected outcomes or outputs
ELT Pipeline Architecture is a concept in architecture. In simple terms, ELT Pipeline Architecture covers architectural patterns for Data Engineering. This data engineering concept addresses key topics in the architectural patterns for data engineering domain. Also known a
Analogy
Example
Find Gaps
Explain ELT Pipeline Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates ELT Pipeline Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for ELT Pipeline Architecture should include: 1. The core components of elt pipeline 2. How they interact 3. Expected outcomes or outputs
Research is a concept in specialized. In simple terms, A concept related to research
Analogy
Example
Find Gaps
Explain Research as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Research in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Research should include: 1. The core components of research 2. How they interact 3. Expected outcomes or outputs