Apache Hudi

What Apache Hudi is for

Apache Hudi is a data lake platform for tables that change. The documentation describes it as bringing database and data warehouse capabilities to the data lake, and the two capabilities it is best known for are record level updates and incremental reads.

Both come from the same design decision. Every record in a Hudi table carries a key, and Hudi keeps an index that maps that key to the file it lives in.

Why record keys change what is possible

A data lake table without keys can only be appended to or rebuilt. If ten rows out of a hundred million need correcting, the usual answer is to rewrite the partitions that contain them.

With a key and an index, Hudi can find the files holding those ten records and act on them directly. An incoming batch becomes an upsert: matching records are updated, the rest are inserted. Deletes work the same way, which matters when a record has to be removed on request.

Choosing a table type

Hudi offers two, and the choice is a trade between write cost and read cost.

Copy on write. Changes are applied when the data is written. The affected files are rewritten so the table always holds finished data. Reads are straightforward, writes do more work.

Merge on read. Changes are written into log files alongside the base files, and combined when the table is read. Writes finish faster, reads do more work, and a background compaction eventually folds the logs into the base files.

Frequent small changes usually point towards merge on read. Tables read far more often than they are written usually point towards copy on write.

The timeline and incremental queries

Hudi records every action against the table on a timeline. Commits, cleaning, compaction and clustering all appear there in order.

This is what makes incremental queries possible. A downstream job records the point it reached, and on the next run asks Hudi for the records committed since that point. Instead of scanning a full table to find what is new, the job reads a much smaller set.

Table services

A lake table left alone accumulates small files and drifting layouts. Hudi treats the fixes as services rather than as jobs you write:

• Compaction merges log files into base files on merge on read tables.
• Clustering reorganises data to improve how queries read it.
• Cleaning removes older file versions that are no longer needed.
• File sizing keeps written files within a sensible range.

Who tends to use Hudi

Hudi suits data engineering teams whose lake tables must reflect changing source systems, particularly where records are captured from a database and applied continuously. It is a platform rather than an end user tool, so it is normally chosen and operated by the team that owns the pipelines.

Things to weigh up

Hudi has more moving parts than a plain file format, because indexes, table services and table types all have to be configured. That configuration is where most of the effort goes.

The table type is also a commitment worth thinking through, since the right answer depends on how often the table changes and how it is read.

Getting started

The documentation includes a quick start guide with Spark and Flink examples that create a table, write to it, run an upsert and then read the table incrementally. There are separate pages covering table types, indexing and each table service.