Designing a Production Data Platform: From Requirements to Architecture
Try This First
Test your knowledge before reading. Don't worry if you get it wrong — that's part of learning.
Key Insights
- Master the full lifecycle of data platform design: requirements gathering, architecture methodology, a fintech case study, data mesh decentralization, open source vs managed decision framework, budget-friendly stacks, and evolving a platform into a data product.
From Requirements to Architecture: A Design Methodology
Designing a production data platform requires a systematic approach that starts with business requirements and ends with a concrete technology architecture. The methodology has five phases:
- Phase 1 — Requirements Gathering: Identify data sources, consumer personas, latency SLAs, volume estimates, compliance constraints (GDPR, SOX, SOC2), and team skill sets. Interview stakeholders across engineering, analytics, ML, and business teams.
- Phase 2 — Logical Architecture: Define the data flow: ingestion → storage → transformation → serving. Choose architectural patterns (medallion, data mesh, lambda/kappa) based on requirements. Map data domains to ownership boundaries.
- Phase 3 — Technology Selection: Evaluate tools against a weighted decision matrix. Factors: team expertise, operational maturity, scale requirements, budget, vendor lock-in risk, and community health.
- Phase 4 — Implementation Plan: Define a phased rollout. Start with a minimal viable platform serving one use case, then add capabilities incrementally. Plan for data migration, schema evolution, and cutover strategies.
- Phase 5 — Operations and Governance: Establish monitoring, alerting, cost governance, data cataloging, and access control. Define SLOs for freshness, quality, and availability. Run a post-mortem after every incident.
Case Study: Building a Data Platform for a Fintech Startup
Let's walk through a realistic scenario. A fintech startup processes 5M transactions/day, needs real-time fraud detection, daily regulatory reporting, and ML-driven credit scoring. Here's how the design unfolds:
| Requirement | Choice | Rationale |
|---|---|---|
| Real-time fraud detection | Kafka + Flink | Sub-second latency, exactly-once semantics, stateful processing |
| Daily regulatory reporting | dbt + Snowflake | SQL-first, tested, auditable transformations |
| ML credit scoring | Feast + Databricks | Feature store for training/serving consistency |
| Orchestration | Dagster | Software-defined assets, dbt integration, partitioned backfills |
| Storage | S3 + Iceberg | Open table format, ACID transactions, schema evolution |
| Data quality | Great Expectations + dbt tests | Expectations at ingestion, tests at transformation |
| Observability | Monte Carlo + OpenLineage | Automated lineage, freshness monitoring, incident management |
Key design decisions: Use a medallion architecture (Bronze for raw, Silver for cleaned, Gold for aggregated) with Iceberg as the table format. Run batch pipelines for regulatory reporting on Snowflake (SQL-friendly for auditors), and streaming pipelines for fraud detection on Flink. Dagster orchestrates both worlds.
Data Mesh: Decentralized Ownership at Scale
As the organization grows beyond a single data team, the centralized platform model breaks down. Data mesh, proposed by Zhamak Dehghani, offers an alternative: domain teams own their data as products, while a shared infrastructure layer provides cross-domain interoperability.
- Domain Ownership: Each business domain (payments, risk, marketing, compliance) owns its data — from ingestion to serving. The domain team decides schema, quality standards, and access policies.
- Data as a Product: Each domain exposes its data as a well-documented, versioned, and discoverable product with SLOs. Consumers subscribe to data products rather than requesting one-off exports.
- Self-Serve Infrastructure: A shared platform team provides the infrastructure (storage, compute, catalog, lineage) that domains use to build and serve their data products. This includes data discovery (DataHub/Amundsen), lineage (OpenLineage), and access control.
- Federated Governance: Global standards (naming conventions, security policies, catalog metadata) are defined centrally but enforced locally. Each domain implements standards within its own data product boundaries.
- dbt Mesh: Enables decentralized dbt projects where each domain owns its models and contracts, while sharing cross-project references via a public model API. This is a practical implementation of the data mesh principle for the transformation layer.
Open Source vs Managed: A Decision Framework
| Factor | Choose Open Source | Choose Managed |
|---|---|---|
| Team size | 5+ platform engineers | < 5 engineers |
| Operating budget | Engineering time > vendor cost | Vendor cost > engineering time |
| Customization needs | High (custom connectors, transforms) | Standard patterns only |
| Compliance requirements | Data residency, air-gapped | SOC2/HIPAA certified vendor |
| Scalability horizon | Predictable, gradual growth | Rapid, unpredictable scaling |
Most organizations use a hybrid approach: managed services for commodity capabilities (data warehouse, message broker) and open source for differentiation (transformation logic, custom connectors, ML pipelines).
Building a Data Platform on a Budget
Not every organization can afford Snowflake + Fivetran + Monte Carlo. Here's a cost-effective open-source stack that scales from startup to mid-market:
- Ingestion: Airbyte (open source) + Kafka (self-managed or Confluent Cloud free tier). Airbyte has 300+ connectors for sources and destinations.
- Storage: S3/MinIO + Apache Iceberg. MinIO provides S3-compatible storage on-premises or in any cloud. Iceberg provides table management, ACID, and time travel.
- Transformation: dbt Core (free) + DuckDB for local development. dbt Cloud is optional for CI/CD and scheduling — use Dagster instead.
- Orchestration: Dagster (open source). The Dagster+ managed service is optional; the OSS version is production-ready.
- Data Quality: Great Expectations (open source) + dbt tests (built into dbt Core). Both are free and widely adopted.
- Observability: OpenLineage (free) + Elementary (free dbt-native observability) + custom monitoring with Prometheus/Grafana.
- BI: Metabase or Apache Superset (both open source). Good enough for internal dashboards and ad-hoc querying.
This stack costs <$500/month for infrastructure at startup scale and can handle terabytes of data. As you grow, you can replace components with managed alternatives without rewriting your pipeline logic — because the core transformation code (dbt models, Python assets) is portable.
From Data Platform to Data Product
The final evolution is treating the data platform itself as a product. This means:
- Developer Experience: Provide self-service tools for data ingestion (web UI or API), transformation templates, and CI/CD pipelines. Data engineers should not be a bottleneck for adding new data sources.
- Documentation and Discovery: Every dataset has a description, owner, freshness SLO, quality score, and lineage. Use a data catalog (DataHub, Amundsen, or Atlan) to make data discoverable.
- Cost Transparency: Show each team their storage and compute costs. Chargeback or showback models incentivize efficient usage. Monitor cost per dataset and alert on anomalies.
- Feedback Loops: Collect feedback from data consumers through surveys, usage analytics, and regular meetings. Treat the platform backlog like any product backlog — prioritize by impact.
The ultimate measure of a data platform is not its technology stack but its time-to-insight: how long does it take a data consumer to go from a question to a trusted answer? A well-designed platform reduces this from weeks to minutes.
Article Metadata
Review with Spaced Repetition
Add this lesson's 7 flashcards to your SM-2 study queue. They will appear when due in the Study Queue.
Feynman Concept Cards
Master each building block: read the ELI5, explore the analogy, work the example, find your gaps, teach it back, build it.
Data Mesh is a concept in architecture. In simple terms, Data Mesh covers architectural patterns for Data Engineering. This data engineering concept addresses key topics in the architectural patterns for data engineering domain. Also known as: data mesh arc
Analogy
Example
Find Gaps
Explain Data Mesh as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Data Mesh in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Data Mesh should include: 1. The core components of data mesh 2. How they interact 3. Expected outcomes or outputs
DataOps is a concept in best practices. In simple terms, DataOps covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: DataOps practices, data operation
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Lakehouse Architecture is a concept in architecture. In simple terms, The Lakehouse architecture, formalized by Armbrust et al. (2021), combines the flexibility of data lakes (cheap object storage, diverse data types) with the reliability of data warehouses (ACID transa
Analogy
Example
Find Gaps
Explain Lakehouse Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Lakehouse Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Lakehouse Architecture should include: 1. The core components of lakehouse architecture 2. How they interact 3. Expected outcomes or outputs
Design is a concept in specialized. In simple terms, A concept related to design
Analogy
Example
Find Gaps
Explain Design as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Design in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Design should include: 1. The core components of design 2. How they interact 3. Expected outcomes or outputs
Feynman Synthesis — Prove You Understand
1. The One-Pager
Explain this lesson's core idea to a smart 15-year-old. No jargon allowed.
2. The Gap Map
List 3 things you are still unsure about. Be specific.
Knowledge Check
Test your understanding of this lesson.
Flashcards
Space = flip · 1-4 = grade · Swipe on mobile