Data Engineering Weekly Notes
May 18 – May 24 · 2026-W21
2 events this week
🏛️ Quiet week Compliance
📈 Warsh's first full week Markets
⚡⚡⚡ ICBERG WINS THE FORMAT WAR Data Engineering
🧠 Feynman Takeaway
Iceberg v3 GA on Snowflake, preview on Databricks. Iceberg v4 development begins. The format war is over — Iceberg is the industry standard.
Iceberg v3 GA on Snowflake, Preview on Databricks
In plain English
Iceberg v3 went GA on Snowflake and entered preview on Databricks. This was the moment the format war ended: both major cloud data platforms now support Iceberg natively. Delta Lake conceded via UniForm (which provides Iceberg compatibility). The industry standard was settled.
Analogy
Imagine VHS vs. Betamax, Blu-ray vs. HD DVD, or iPhone vs. Android. Every format war ends when the market picks one. Iceberg won. Not by being clearly better in every dimension, but by being the most open and most widely supported.
Why it matters
This had enormous implications: tool vendors could invest in Iceberg support without hedging, data teams could standardize on one format, and the ecosystem could focus on innovation rather than compatibility.
Iceberg v4 Development Initiated
In plain English
The Iceberg community began the v4 development process: relative paths for cloud-agnostic table references, content statistics for query optimization, and enhanced metadata management. The standard was already moving to the next generation.
Analogy
No sooner had Iceberg won the format war than the community started building the next version. It's like the Olympic marathon winner immediately starting to train for the next race.
Why it matters
Iceberg v4's relative paths feature was particularly important: it would allow Iceberg tables to be referenced in a cloud-agnostic way, further reducing vendor lock-in.