Data Quality Engineering: Testing, Monitoring, and Expectations at Scale
Try This First
Test your knowledge before reading. Don't worry if you get it wrong — that's part of learning.
Key Insights
- Why Data Quality Engineering Matters In a DataOps culture, data quality is not discovered after the fact — it is engineered into the pipeline from day one.
- Every time data moves between st.
Why Data Quality Engineering Matters
In a DataOps culture, data quality is not discovered after the fact — it is engineered into the pipeline from day one. Every time data moves between stages, there is a risk of corruption, drift, or silent failure. Data quality engineering applies software testing principles to data.
Three Layers of Data Quality
1. Freshness
Is the data current? A pipeline SLO might state: "All source data must be ingested within 5 minutes of production." Violations trigger alerts and page the on-call data engineer. Freshness checks compare the max timestamp of incoming records against the current time.
2. Completeness
Are all expected records present? Row count checks compare expected vs actual counts. A sudden drop of 30% in event volume usually means a source connector failed or schema changed.
3. Accuracy
Do the values make sense? Accuracy checks include: NULL ratio thresholds, reference integrity (every foreign key has a matching primary key), domain constraints (age between 0 and 120), and statistical distribution checks (z-score outlier detection).
Expectations as Code
Great Expectations lets you define expectations as Python objects:
expectation_suite = ExpectationSuite("transactions_clean")
expectation_suite.add_expectation(
ExpectColumnValuesToBeBetween("amount", 0, 1000000)
)
expectation_suite.add_expectation(
ExpectColumnValuesToNotBeNull("transaction_id")
)
These expectations run on every pipeline execution. If they fail, the pipeline either halts (hard gate) or proceeds with a warning (soft gate) depending on severity.
Monitoring Dashboard
A production data quality dashboard tracks:
- Pass/fail rate per expectation over time
- Data freshness (minutes since last successful ingestion) per source
- Row count trends with anomaly detection (sudden drops/spikes)
- Schema change events (new columns, dropped columns, type changes)
- Pipeline DAG health (success rate, duration, backlog)
The AcaciaFund SQI (Signal Quality Index) is itself a data quality metric — combining source authority, freshness, consensus, and relevance into a single composable score.
Article Metadata
Review with Spaced Repetition
Add this lesson's 4 flashcards to your SM-2 study queue. They will appear when due in the Study Queue.
Feynman Concept Cards
Master each building block: read the ELI5, explore the analogy, work the example, find your gaps, teach it back, build it.
Data Quality is a concept in best practices. In simple terms, Data Quality covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: data observability, data val
Analogy
Example
Find Gaps
Explain Data Quality as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates Data Quality in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for Data Quality should include: 1. The core components of data quality 2. How they interact 3. Expected outcomes or outputs
DataOps is a concept in best practices. In simple terms, DataOps covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: DataOps practices, data operation
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Data Observability is a concept in best practices. In simple terms, Data Observability covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: data monitoring, data
Analogy
Example
Find Gaps
Explain Data Observability as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates Data Observability in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for Data Observability should include: 1. The core components of data observability 2. How they interact 3. Expected outcomes or outputs
Feynman Synthesis — Prove You Understand
1. The One-Pager
Explain this lesson's core idea to a smart 15-year-old. No jargon allowed.
2. The Gap Map
List 3 things you are still unsure about. Be specific.
Knowledge Check
Test your understanding of this lesson.
Flashcards
Space = flip · 1-4 = grade · Swipe on mobile