Data Quality: The Foundation of Trustworthy Pipelines

Data quality is not a one-time exercise — it is an ongoing practice embedded into every stage of the data pipeline. Poor data quality costs organizations an estimated 15-25% of revenue and erodes trust in data-driven decision making.

• Completeness: Are all required fields present? Missing values can indicate source system issues or pipeline failures.
• Uniqueness: Are there duplicate records? Duplicate detection prevents double-counting in aggregations and reporting.
• Timeliness: Is the data fresh enough for its intended use? A dashboard that refreshes hourly needs different freshness guarantees than a monthly regulatory report.
• Validity: Does the data conform to expected formats, types, and ranges? Invalid emails, negative ages, or future dates signal upstream problems.
• Accuracy: Does the data reflect reality? This is the hardest dimension to measure — it often requires cross-referencing with trusted sources or manual sampling.
• Consistency: Does the data agree across systems? The same customer name or product code should match across source, warehouse, and BI layer.

Great Expectations: Data Quality as Code

Great Expectations (GX) is the most widely adopted open-source data quality framework. It lets teams define, document, and test data expectations programmatically.

• Expectations: The core abstraction. An expectation is a declarative, version-controlled statement about your data (e.g., expect_column_values_to_not_be_null("order_id") or expect_column_sum_to_be_between("revenue", 100000, 500000)).
• Data Docs: Auto-generated HTML documentation showing expectation results, data samples, and pipeline health. Teams can browse data quality reports without writing code.
• Suites and Checkpoints: An expectation suite groups related expectations for a dataset. A checkpoint runs a suite against a specific batch of data and produces validation results.
• Integration: GX integrates with dbt (run expectations on dbt models), Airflow/Dagster (as a pipeline step), and most data warehouses via SQLAlchemy or Spark.

A typical Great Expectations workflow:

import great_expectations as gx

context = gx.get_context()
datasource = context.sources.add_spark("my_spark")
data_asset = datasource.add_csv_asset("orders", "s3://data/orders/")
batch = data_asset.get_batch()

Define expectations
batch.expect_column_to_exist("order_id")
batch.expect_column_values_to_not_be_null("amount")
batch.expect_column_values_to_be_between("amount", 0, 100000)
batch.expect_column_mean_to_be_between("amount", 50, 200)

Validate
results = batch.validate()
print(f"Passed: {results['statistics']['successful_expectations']}")

Data Observability: Monitoring the Pipeline Health

Data observability extends monitoring to the data itself — not just the infrastructure running the pipeline. It answers five key questions:

• Freshness: Is the data arriving on time? Track scheduled vs actual arrival times for each table or topic.
• Volume: Is the expected amount of data arriving? Sudden drops or spikes indicate upstream issues or schema changes.
• Schema: Has the data structure changed? New columns, removed columns, or type changes must be detected automatically.
• Quality: Are data quality expectations passing? Track pass/fail rates over time and alert on degradation.
• Lineage: Where did this data come from, and what downstream reports does it feed? Full column-level lineage enables impact analysis.

Key tools in the observability space:

Cost Optimization: Engineering for Efficiency

Data infrastructure costs grow with data volume. Without deliberate optimization, cloud data costs can spiral 30-50% year over year. Here are the key strategies:

• Partition Pruning: Organize tables by date, region, or other high-cardinality dimensions. Queries that filter on the partition key only scan relevant partitions, reducing cost and improving speed. Tools like Iceberg and Delta Lake support partition evolution — changing the partition scheme without rewriting data.
• Incremental Processing: Process only new or changed records instead of full-table scans. dbt incremental models, Spark structured streaming with watermarks, and Kafka compacted topics all reduce per-run compute.
• Storage Tiering: Move cold data to cheaper storage. Use lifecycle policies to transition data from hot (SSD) to warm (HDD) to cold (object store/Glacier) based on access patterns. Iceberg and Delta Lake support storage tiering transparently.
• Compute Autoscaling: Use serverless or auto-scaling compute to match resources to workload. Snowflake auto-suspend, BigQuery slot reservations, and Spark dynamic allocation prevent over-provisioning.
• Query Optimization: Identify and fix expensive queries. Use materialized views for frequent aggregations, sort keys/partitioning for common filter patterns, and avoid SELECT * in production pipelines.
• Data Retention and Purging: Define retention policies per dataset. Raw source data may be retained for 30 days, transformed data for 6 months, and aggregated data indefinitely. Automate purging to prevent storage bloat.

Feature Stores: Quality-Controlled ML Data

Feature stores bridge the gap between data engineering and ML. They provide a centralized registry of features — pre-computed, versioned, and quality-controlled — that ML models consume in both training and inference.

• Feast: Open-source feature store (LF AI & Data Foundation). Features are defined as Python functions or dbt models, stored in an online store (Redis, DynamoDB) for low-latency serving and an offline store (BigQuery, Snowflake) for training.
• Tecton: Enterprise feature platform built on Feast's architecture. Adds automated feature engineering, streaming feature computation, point-in-time correctness guarantees, and governance controls.
• Data Quality Integration: Feature stores enforce quality at the feature level — detecting drift between training and serving data, validating feature value ranges, and alerting on stale or missing features.

Building an Observability-Driven Quality Program

A production-grade data quality program combines multiple layers:

Layer 1: Source Validation (Great Expectations)
- Check completeness, uniqueness, schema at ingestion
- Alert on anomalies before they enter the pipeline

Layer 2: Transformation Quality (dbt tests, Soda)
- Test assumptions after each transformation stage
- Enforce contracts between model layers

Layer 3: Pipeline Health (Dagster + Elementary)
- Track freshness, volume, and run status per asset
- PagerDuty/Slack alerts on failures

Layer 4: Cross-System Observability (Monte Carlo / OpenLineage)
- Column-level lineage from source to dashboard
- Impact analysis for schema changes
- Incident management and root cause analysis

Layer 5: Cost Governance (Budgets + Alerts)
- Track compute and storage cost per team/dataset
- Set budgets and auto-alert on >10% cost increase
- Quarterly review of unused / cold data

The goal is not 100% data quality — that is neither achievable nor cost-effective. Instead, define quality SLOs for each data product based on its criticality. A board-level financial report needs higher quality guarantees than an internal experimentation dataset.