Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lakehouse table format that provides ACID transactions, record-level updates/deletes, incremental processing, and streaming ingestion on distributed file systems (HDFS, S3, GCS, Azure ADLS). It sits atop Parquet/Avro files and integrates with Spark, Flink, Trino, Hive, Presto, and cloud-native catalog services (Glue, Unity Catalog, Polaris). This cheat sheet covers all core concepts from table types and write operations through indexing, compaction, clustering, schema evolution, and multi-engine integrations.
What This Cheat Sheet Covers
This topic spans 17 focused tables and 284 indexed concepts, 115 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Core Architecture Concepts
Hudi organises data into file groups within partitions. Each file group contains ordered file slices consisting of a base file (Parquet) and optional delta/log files (Avro or Parquet). The timeline — a log of all table actions stored in .hoodie/ — provides MVCC snapshot isolation so readers never see partial writes. The metadata table (a hidden Hudi table itself) replaces expensive file-listing calls with indexed lookups.
| Concept | Example | Description | |
|---|---|---|---|
partition/.hoodie/<fileId>/ | • Logical unit for a set of records sharing the same fileId• each file group maps to one record-key range | ||
base file + 0..N log files at one instant | • Versioned snapshot of a file group at a commit instant • latest slice = current data | ||
part-0000_<fileId>_<commitTime>.parquet | • Columnar Parquet file containing full row data • written at compaction (MOR) or every commit (COW) | ||
.<fileId>_<commitTime>.log.1 | • Append-only file storing delta records (inserts, updates, deletes) for MOR tables • read-merged at query time | ||
.hoodie/metadata/ | • Hidden Hudi MOR table storing files, column_stats, partition_stats, bloom_filters, rli indexes• eliminates O(N) file listings | ||
.hoodie/<timestamp>.<action>.<state> | • Ordered log of all table actions • provides MVCC and time-travel anchors • backed by LSM-style storage | ||
20240615120000000.commit.completed | • Single unit of the timeline • 3-state lifecycle: requested → inflight → completed | ||
timestamp = action completion clock time | • Instants are assigned monotonically increasing millisecond timestamps • prevents out-of-order visibility | ||
snapshot isolation | • Readers latch on the latest completed instant • writers create a new instant • no read-write conflicts | ||
year=2024/month=06/day=15 | • Hive-style or custom partition under the table base path • combined with record key = unique record address |