Compliance-by-Design: When the Rulebook Lives in the Pipeline
Key Insights
- Why compliance is becoming an architecture problem: rules as versioned code, data contracts as legal interfaces, lineage as evidence, and circuit breakers — with a deliberate human residue for contestability.
Compliance is increasingly an architecture problem, not a policy problem. If a rule must be enforced, the strongest way to enforce it is to make the compliant path the only structurally possible path. This essay lays out compliance-by-design: rules as versioned code, data contracts as legal interfaces, lineage as evidence, and circuit breakers as the last line of defence.
From inspection to design
There are two ways to achieve compliance. Ex-post inspection lets behaviour happen and then checks whether it broke a rule — audits, reviews, fines. It is reactive, sampled, and always one step behind. Ex-ante design embeds the rule in the machinery so that violating behaviour is difficult or impossible to produce in the first place — controls, permission systems, schema constraints, real-time screening.
Lessig's famous claim that "code is law" cuts both ways here: if software architecture regulates behaviour as effectively as statutes, then architecture is also where compliance should be built. Design is not a softening of law — it is law's most reliable enforcement mechanism.
Rules as versioned artifacts
The core move of compliance-by-design is to treat a rule as an artifact: a versioned, tested, deployable unit, managed the way software is managed. This is policy-as-code — the pattern familiar from Open Policy Agent and Kubernetes admission controllers, applied to compliance domains.
- Versioning: every rule change is a commit with an author, a timestamp, and a rationale. The rule history is the legislative record.
- Testing: each rule carries fixtures — the cases it must catch and the cases it must not. A rule that cannot be tested cannot be trusted.
- Auditability: because the rule is executable, you can prove which version was in force at any moment, and exactly what it did.
In this framing, Git is a legislature: it provides recognition, change, and a traceable record of every decision — Hart's secondary rules, implemented in a version-control system.
Data contracts as legal interfaces
A data contract is a binding agreement between two systems about the shape, semantics, and provenance of the data they exchange. It is the data-engineering analogue of a treaty: it fixes what words mean, what is required, and what happens when either side changes.
For compliance-by-design, contracts are where legal meaning becomes machine-checkable. When a contract says a field "must be verified", that is a legal obligation encoded as a schema constraint. When lineage records every transformation a value passed through, that is an evidentiary trail a regulator can audit. Contracts give the law layer a reliable view of the information substrate — and without that view, the law layer is blind.
Evidence as lineage
Enforcement depends on evidence, and in a data-driven system evidence is provenance. Data lineage — who produced a value, from what inputs, through which transformations — converts an assertion ("we screened this customer") into a claim that can be verified end to end.
This is why observability is a compliance capability: a pipeline you cannot trace is a compliance process you cannot defend. The same logs that debug an outage document the case file.
Circuit breakers and the human residue
Even the best design fails, so the system needs circuit breakers: kill switches that halt or quarantine a process when it crosses a threshold — a runaway model, a suspicious burst of activity, a data feed gone stale. Circuit breakers are the admission that architecture cannot foresee everything, and that the system must fail toward caution rather than toward exposure.
Equally important is the human residue. Not everything should be automated. Open texture (Hart), contestability, and discretion all argue for keeping a human review lane in the loop — especially for decisions that affect a person's access to services. Compliance-by-design is not automation-at-all-costs; it is architecture that decides which decisions are safe to automate and which must remain contestable by a person.
Article Metadata
Bloom Taxonomy Questions
What is the difference between ex-post inspection and ex-ante design as compliance strategies?
Compare a sanctions-screening rule enforced at the database level versus one applied by a downstream reviewer. What changes in risk and evidence?
Where must a human remain in the loop in an automated compliance system, and why?
Sketch a policy-as-code rule for a KYC requirement, including the fixtures that would test it.
Further Reading
Databricks Blog
Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog
Apache Kafka
Kafka documentation, KIPs, and ecosystem updates
Apache Flink
Flink documentation and release notes
Apache Iceberg
Iceberg table format — specs, REST catalog, performance
dbt Blog
dbt Labs engineering blog — analytics engineering, Semantic Layer
Dagster Blog
Dagster orchestration — software-defined assets, IO managers
Feynman Concept Cards
Master each concept: read the ELI5, explore analogies, work examples, and teach it back.
Data Contracts is a concept in architecture. In simple terms, Data Contracts covers architectural patterns for Data Engineering. This data engineering concept addresses key topics in the architectural patterns for data engineering domain. Also known as: data con
Analogy
Example
Find Gaps
Explain Data Contracts as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Data Contracts in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Data Contracts should include: 1. The core components of data contracts 2. How they interact 3. Expected outcomes or outputs
Compliance-by-Design is a concept in regtech. In simple terms, The practice of embedding legal and regulatory rules directly into the architecture of data pipelines and software systems — versioned rule artifacts, data contracts as legal interfaces, lineage as ev
Analogy
Example
Find Gaps
Explain Compliance-by-Design as if teaching a colleague who is new to regtech. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Compliance-by-Design in a real-world regtech scenario. Walk through your design decisions.
Show solution
A diagram for Compliance-by-Design should include: 1. The core components of compliance by design 2. How they interact 3. Expected outcomes or outputs
Design is a concept in specialized. In simple terms, A concept related to design
Analogy
Example
Find Gaps
Explain Design as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Design in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Design should include: 1. The core components of design 2. How they interact 3. Expected outcomes or outputs