Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms
Key Insights
- The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.
Edit on GitHub — registry.json
Background
For a decade, data teams chose between two unhappy options. Data lakes stored anything in open formats at low cost but offered no transactions, no schema enforcement, and weak performance. Data warehouses delivered ACID guarantees and fast SQL but were proprietary, expensive, and siloed. Armbrust and colleagues at Databricks argued the choice was false: the two could be unified into a single open platform called the lakehouse.
The Architecture
A lakehouse keeps data in open file formats (Parquet, ORC, Avro) on cloud object storage, and adds a transaction layer over the files: metadata that records which files belong to which table version. This is exactly what open table formats such as Delta Lake, Apache Iceberg, and Hudi implement, and why they are the load-bearing components of the architecture.
Deep Dive
The transaction layer provides the guarantees warehouses always had: ACID transactions through optimistic concurrency on the table metadata, snapshot isolation so readers never see partial writes, and time travel via retained versions. Schema enforcement and evolution become metadata operations rather than full rewrites. Because the data itself is engine-agnostic, Spark, Presto/Trino, Flink, and DuckDB can all query the same tables. The paper also positions machine learning workloads — which want direct file access and feature stores — as native beneficiaries, since warehouses could never serve them.
Why It Matters
The lakehouse is now the default architectural pattern for modern data platforms. The paper's central claim — that the delta between lake and warehouse is a metadata layer, not a new engine — explains the consolidation of the data stack around open formats and catalogs.
Key Takeaways
- Choose the table format deliberately; it determines your transaction, schema, and partitioning semantics.
- Multi-engine access is the point — keep data open, not locked to one engine's catalog.
- Object storage still demands file-level hygiene: compaction and snapshot management remain operational duties.