Building Resilient Data Pipelines with Spark and dbt

Architecture: Spark + dbt

The modern data stack combines Spark for heavy ETL processing with dbt for SQL-based transformations, testing, and documentation:

• Spark — ingestion, heavy computation, ML feature engineering
• dbt — business logic transformations, data quality tests, documentation
• Orchestration — Dagster/Airflow coordinates Spark jobs → dbt runs

# Spark: heavy lifting (ingestion, joins, aggregation)
# dbt/models/staging/stg_transactions.sql
WITH source AS (
 SELECT * FROM {{ source('raw', 'transactions') }}
),
cleaned AS (
 SELECT
 transaction_id,
 customer_id,
 CAST(amount AS DECIMAL(18,2)) AS amount,
 UPPER(currency) AS currency,
 DATE(transaction_ts) AS transaction_date,
 CASE 
 WHEN amount > 10000 THEN 'high_value'
 WHEN amount > 1000 THEN 'medium'
 ELSE 'standard'
 END AS value_tier
 FROM source
 WHERE transaction_id IS NOT NULL
)
SELECT * FROM cleaned

Data Quality Tests

dbt tests validate your data at every transformation step:

# dbt/models/staging/schema.yml
version: 2
models:
 - name: stg_transactions
 columns:
 - name: transaction_id
 tests:
 - unique
 - not_null
 - name: amount
 tests:
 - not_null
 - dbt_utils.accepted_range:
 min_value: 0
 max_value: 10000000
 - name: currency
 tests:
 - accepted_values:
 values: ['USD', 'EUR', 'GBP', 'JPY']

Lineage and Documentation

dbt auto-generates lineage graphs and documentation. Combined with Spark's execution plans, you get end-to-end visibility:

• Column-level lineage — trace any metric back to its raw source
• Impact analysis — understand downstream effects of schema changes
• Run history — track freshness, execution time, and test results