Data Lakehouse Cheat Sheet

Updated 2026-04-21

Next Topic: Data Mesh Architecture Cheat Sheet

🧠Study flashcards on this topic146 cards · spaced repetition→

A data lakehouse is a modern data architecture that unifies the scalability of data lakes with the reliability of data warehouses by layering open table formats (Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon) on top of low-cost cloud object storage. The architecture enforces ACID transactions, schema evolution, and governance while keeping compute and storage fully decoupled — enabling SQL analytics, real-time streaming, and ML workloads to operate on a single copy of data without duplication. By 2026, the lakehouse model has matured from experimental to mainstream: the Iceberg REST Catalog has become the vendor-neutral standard, Iceberg V3 adds deletion vectors and row lineage, Delta Lake 4.0 brings Liquid Clustering and Coordinated Commits, and newer entrants like DuckLake and Lance are challenging traditional metadata architectures and serving AI-native workloads.

What This Cheat Sheet Covers

This topic spans 19 focused tables and 158 indexed concepts, 146 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core ConceptsTable 2: Open Table FormatsTable 3: File FormatsTable 4: Metadata CatalogsTable 5: Architecture PatternsTable 6: Storage Optimization TechniquesTable 7: ACID Transaction FeaturesTable 8: Schema ManagementTable 9: Data Ingestion PatternsTable 10: Streaming IntegrationTable 11: Query EnginesTable 12: Query OptimizationTable 13: Table Maintenance OperationsTable 14: Governance & SecurityTable 15: Cross-Format InteroperabilityTable 16: Lakehouse PlatformsTable 17: Python & Lakehouse ToolsTable 18: Use Cases & WorkloadsTable 19: Benefits & Challenges

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Concepts

Start here for the vocabulary that everything else builds on. These are the ideas that turn cheap object storage into something that behaves like a database—open table formats, ACID transactions, time travel, schema evolution, and the medallion layering that organizes raw data into something analysts can trust.

Concept	Example	Description
Data Lakehouse	Databricks Lakehouse Platform	Unified architecture combining data lake flexibility with data warehouse reliability — supports all data types, ACID transactions, and BI/ML workloads on one platform.
Open Table Format	Iceberg, Delta Lake, Hudi, Paimon	Metadata layer atop object storage providing database-like capabilities — transforms raw files into transactional, versioned, queryable tables.
Compute-Storage Separation	S3 storage + Spark/Trino compute	Decoupling storage (cheap object store) from compute (elastic engines) — multiple engines query the same data independently without duplication.
File-Level Tracking	Manifests listing every data file	Modern table formats track individual files in metadata rather than scanning directories — enables atomic commits, fast planning, and time travel.
ACID Transactions	`MERGE INTO users USING updates ...`	Atomicity, consistency, isolation, durability guarantees for concurrent reads/writes — implemented via transaction logs and optimistic concurrency.
Medallion Architecture	Bronze → Silver → Gold	Data design pattern organizing lakehouse into Bronze (raw), Silver (cleansed), and Gold (curated) layers — incremental quality improvement from ingestion to analytics.
Time Travel	`SELECT * FROM table VERSION AS OF 10`	Query historical snapshots at specific versions or timestamps — enables auditing, rollback, and reproducibility without duplicating data.
Schema Evolution	Add, drop, or rename columns	Modify table schema without rewriting data — new columns appear as NULL in old files, merged on read using column IDs.
Data Skipping	Min/max statistics per file	Optimization using file-level statistics to skip irrelevant files during queries — drastically reduces I/O by pruning files based on predicates.
Compaction	`OPTIMIZE table`	Process of combining small files into larger, optimally-sized files — improves read performance and reduces metadata overhead.
Change Data Capture (CDC)	Debezium → Delta Lake / Iceberg	Pattern for capturing real-time database changes (inserts, updates, deletes) and replicating them to lakehouse tables.
Upsert (MERGE)	`MERGE INTO target USING source ON key` `WHEN MATCHED THEN UPDATE` `WHEN NOT MATCHED THEN INSERT`	Update existing rows and insert new ones based on match condition — essential for SCD Type 1 patterns and CDC ingestion.

Table 2: Open Table Formats

The open table format is the beating heart of the lakehouse—the metadata layer that makes a pile of Parquet files act like a transactional table. Iceberg, Delta Lake, Hudi, and Paimon are the four established players, each with its own strengths, and the newer entrants DuckLake and Lance show where the format wars are heading in 2026.

Format	Example	Description
Apache Iceberg	`CREATE TABLE t USING iceberg`	Vendor-neutral open table format governed by the Apache Foundation — hidden partitioning, partition evolution, snapshot isolation, and the broadest multi-engine support make it the 2026 default.

The rest of this section is part of the full cheat sheet

Table 3: File Formats

Underneath the table format, the actual bytes live in a file format—and the choice still matters for compression, query speed, and write patterns. Parquet is the near-universal default for columnar analytics, ORC and Avro fill specific niches, and Lance is the AI-native newcomer built for embeddings and multimodal data.

Format	Example	Description
Parquet	Columnar binary format	• Column-oriented format with strong compression and predicate pushdown — default for Delta Lake, Iceberg, and Hudi • the universal lakehouse data file format.

The rest of this section is part of the full cheat sheet

Table 4: Metadata Catalogs

A catalog is what lets many engines find and govern the same tables without stepping on each other, and in 2026 the Iceberg REST Catalog spec has made this layer the key to avoiding vendor lock-in. The options here span managed offerings like Unity Catalog and the open, vendor-neutral Polaris, down to lightweight self-hosted choices and the legacy Hive Metastore most teams are migrating away from.

Catalog	Example	Description
Unity Catalog	Databricks-managed catalog	Centralized governance for data and AI assets — fine-grained access control, lineage, multi-format (Delta, Iceberg via REST/UniForm), open-sourced.

The rest of this section is part of the full cheat sheet

Table 1: Core Concepts

Concept	Example	Description
Data Lakehouse	Databricks Lakehouse Platform	Unified architecture combining data lake flexibility with data warehouse reliability — supports all data types, ACID transactions, and BI/ML workloads on one platform.
Open Table Format	Iceberg, Delta Lake, Hudi, Paimon	Metadata layer atop object storage providing database-like capabilities — transforms raw files into transactional, versioned, queryable tables.
Compute-Storage Separation	S3 storage + Spark/Trino compute	Decoupling storage (cheap object store) from compute (elastic engines) — multiple engines query the same data independently without duplication.
File-Level Tracking	Manifests listing every data file	Modern table formats track individual files in metadata rather than scanning directories — enables atomic commits, fast planning, and time travel.
ACID Transactions	`MERGE INTO users USING updates ...`	Atomicity, consistency, isolation, durability guarantees for concurrent reads/writes — implemented via transaction logs and optimistic concurrency.
Medallion Architecture	Bronze → Silver → Gold	Data design pattern organizing lakehouse into Bronze (raw), Silver (cleansed), and Gold (curated) layers — incremental quality improvement from ingestion to analytics.
Time Travel	`SELECT * FROM table VERSION AS OF 10`	Query historical snapshots at specific versions or timestamps — enables auditing, rollback, and reproducibility without duplicating data.
Schema Evolution	Add, drop, or rename columns	Modify table schema without rewriting data — new columns appear as NULL in old files, merged on read using column IDs.
Data Skipping	Min/max statistics per file	Optimization using file-level statistics to skip irrelevant files during queries — drastically reduces I/O by pruning files based on predicates.
Compaction	`OPTIMIZE table`	Process of combining small files into larger, optimally-sized files — improves read performance and reduces metadata overhead.
Change Data Capture (CDC)	Debezium → Delta Lake / Iceberg	Pattern for capturing real-time database changes (inserts, updates, deletes) and replicating them to lakehouse tables.
Upsert (MERGE)	`MERGE INTO target USING source ON key` `WHEN MATCHED THEN UPDATE` `WHEN NOT MATCHED THEN INSERT`	Update existing rows and insert new ones based on match condition — essential for SCD Type 1 patterns and CDC ingestion.

Table 2: Open Table Formats

Format	Example	Description
Apache Iceberg	`CREATE TABLE t USING iceberg`	Vendor-neutral open table format governed by the Apache Foundation — hidden partitioning, partition evolution, snapshot isolation, and the broadest multi-engine support make it the 2026 default.

Table 3: File Formats

Format	Example	Description
Parquet	Columnar binary format	• Column-oriented format with strong compression and predicate pushdown — default for Delta Lake, Iceberg, and Hudi • the universal lakehouse data file format.

Table 4: Metadata Catalogs

Catalog	Example	Description
Unity Catalog	Databricks-managed catalog	Centralized governance for data and AI assets — fine-grained access control, lineage, multi-format (Delta, Iceberg via REST/UniForm), open-sourced.