What is dbt?

dbt (data build tool) is the industry-standard SQL-first transformation framework. It enables analytics engineers and data analysts to define data transformations as version-controlled SQL SELECT statements. dbt handles the T in ELT — transforming data that is already loaded into your warehouse or lakehouse.

• Models: The core unit of dbt. Each model is a .sql file containing a SELECT statement. dbt compiles and executes these against your warehouse, materializing the results as tables, views, or incremental models.
• Tests: Built-in and custom data quality tests. Generic tests (unique, not_null, accepted_values, relationships) are defined in YAML. Custom tests are written as SQL SELECT statements that return failing rows.
• Documentation: dbt auto-generates documentation from model definitions and inline description blocks. The docs site includes lineage graphs (Lineage DAG), column-level descriptions, and test coverage.
• Sources and Freshness: Define source tables with freshness assertions. dbt tracks when each source was last loaded and alerts if data is stale.
• Materializations:table (full refresh), view (no storage), incremental (append/merge new records), ephemeral (CTE), and materialized_view (database-native).
• dbt Mesh: Decentralize dbt projects across domain teams while sharing a common cross-project reference layer. Each domain owns its models and contracts.

A simple dbt model looks like this:

-- models/orders_summary.sql
{{ config(materialized='incremental', unique_key='order_id') }}

SELECT
 o.order_id,
 o.customer_id,
 o.order_date,
 o.total_amount,
 c.customer_segment
FROM {{ source('ecommerce', 'orders') }} o
LEFT JOIN {{ ref('customer_segments') }} c
 ON o.customer_id = c.customer_id
{% if is_incremental() %}
 WHERE o.order_date >= (SELECT max(order_date) FROM {{ this }})
{% endif %}

What is Dagster?

Dagster is a next-generation data orchestrator designed for the modern data stack. Unlike traditional orchestrators (Airflow) that treat pipelines as task DAGs, Dagster treats them as software-defined assets — each asset knows its upstream dependencies and how it is materialized.

• Assets: The core abstraction. An asset is a data product (table, file, ML model) with a known producer function. Assets form a DAG based on their dependencies.
• Ops and Jobs: Lower-level building blocks. An op is a single computation. A job is a graph of ops. Assets are built on top of ops.
• Schedules and Sensors: Schedules trigger jobs on a cron interval (e.g., daily at 6 AM). Sensors trigger jobs based on external events (e.g., a file landing in S3).
• dbt Integration: Dagster has a native dbt integration (dagster-dbt) that loads your dbt models as Dagster assets. Dagster orchestrates dbt runs, provides asset-level lineage, and enables cross-tool dependencies (e.g., "run dbt model A, then run Python op B, then run dbt model C").
• Software-Defined Assets (SDA): Assets are declared in Python with their upstream dependencies, partition definitions, and freshness policies. Dagster materializes only what's stale or upstream of a changed asset.
• Observability: Dagster's Dagit UI shows asset lineage, run history, logs, and alerts. Each asset has a "last materialized" timestamp, upstream/downstream lineage, and launch history.

Building a Pipeline with dbt + Dagster

The modern data stack often pairs dbt for transformations with Dagster for orchestration. Here's how they work together:

definitions.py — Dagster project root
from dagster import Definitions, load_assets_from_modules
from dagster_dbt import dbt_assets, DbtCliResource
from pathlib import Path

@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context, dbt: DbtCliResource):
 yield from dbt.cli(["build"], context=context).stream()

defs = Definitions(assets=[my_dbt_assets])

This single file loads your entire dbt project as Dagster assets. Dagster handles scheduling, alerting, and lineage — dbt handles the actual SQL transformations.

A Complete Pipeline Example

Here's a practical pattern combining dbt, Dagster, and Python for a complete data pipeline:

assets.py
from dagster import asset, AssetIn
import pandas as pd

@asset
def raw_transactions() -> pd.DataFrame:
 """Extract: load raw data from source"""
 return pd.read_parquet("s3://landing/transactions/")

@asset
def cleaned_transactions(raw_transactions: pd.DataFrame) -> pd.DataFrame:
 """Transform: clean and validate"""
 df = raw_transactions.dropna(subset=["amount", "timestamp"])
 df["amount"] = df["amount"].clip(lower=0)
 return df

dbt models handle the SQL transformations (aggregations, joins, etc.)
defined in models/ directory as .sql files
loaded via @dbt_assets(...)

@asset
def alerts(cleaned_transactions: pd.DataFrame) -> None:
 """Serve: check thresholds and send alerts"""
 high_value = cleaned_transactions[
 cleaned_transactions["amount"] > 10_000
 ]
 if len(high_value) > 0:
 print(f"Alert: {len(high_value)} high-value transactions detected")

Dagster's asset graph automatically determines execution order: raw_transactions → cleaned_transactions → dbt models → alerts. If a partition of raw_transactions is reprocessed, only downstream assets for that partition are re-materialized.

Key Patterns and Best Practices

• Start with dbt-first: Use dbt for all SQL transformations. Only bring in custom Python when SQL is insufficient (e.g., ML inference, API calls, complex business logic).
• Use incremental models for large tables: dbt's incremental materialization appends or merges only new/changed records, avoiding full table scans.
• Define data contracts: Use dbt's contract enforcement to guarantee column names, types, and constraints between dbt models.
• Leverage Dagster partitions: Partition your assets by date, region, or other dimensions. Dagster tracks which partitions are materialized and can materialize only stale partitions.
• Automate backfills: When upstream source data is corrected, Dagster's "re-materialize upstream" feature cascades the backfill through the entire dependency graph.
• Test in CI: Run dbt build --select state:modified in CI to test only changed models. Use dbt source freshness to validate data recency.

Comparing the Orchestrators

Dagster's software-defined asset approach is a paradigm shift from traditional orchestrators. It treats data as the central concern, not task execution. For teams building modern data platforms, Dagster + dbt is becoming the default stack.