Apache Iceberg
An open table format that lets several engines read and write the same large analytic tables safely.
Apache Iceberg is a table format for very large analytic tables held in object storage. It keeps a set of metadata files that describe exactly which data files belong to a table at any point in time. That record is what allows engines such as Spark, Flink and Trino to work with one table together without corrupting it.
Apache Iceberg in one paragraph
Apache Iceberg is an open table format for very large analytic tables. It does not replace your storage and it does not process queries. It keeps a careful record of which files belong to a table, so that anything reading the table knows exactly what to open.
The documentation describes the goal as bringing the reliability and simplicity of SQL tables to big data, while letting engines such as Spark, Trino, Flink, Presto, Hive and Impala safely work with the same tables at the same time.
Why a table format is needed at all
A folder of Parquet files in object storage is not a table. Nothing records which files are current, which were part of a failed job or which were added while somebody was halfway through a query. Two jobs writing at once can leave a reader with a mix of old and new files.
Iceberg fixes this by writing metadata. Each change produces a new snapshot that lists the files in the table at that moment. Readers pick up a snapshot and stay on it, so a write happening at the same time cannot change the result underneath them.
The pieces Iceberg keeps
- A metadata file that describes the table, its schema and its snapshots.
- A manifest list for each snapshot, pointing at the manifests in that version.
- Manifest files that list the data files themselves, with statistics about the values inside them.
- A catalog entry that points to the current metadata file.
Those statistics matter, because they let an engine skip files that cannot contain matching rows before opening any of them.
What you can do with an Iceberg table
Schema changes are handled as metadata operations. Adding, dropping, renaming or reordering a column does not require rewriting existing data files, and the documentation notes that these changes have no side effects on the values already stored.
Partitioning is handled the same way. Iceberg records the rule that turns a column into a partition value, so a query filters on the ordinary column and Iceberg works out which partitions to read. If the layout turns out to be wrong, partition evolution lets new data use a new layout while existing data stays as it is.
Because every snapshot is retained until it is expired, you can query the table as it stood earlier, or roll the table back to a previous snapshot after a bad write.
Who Iceberg is aimed at
Iceberg suits teams running analytic data in object storage, especially where more than one engine touches the same tables. It is a format rather than an application, so it is normally adopted by data platform and data engineering teams rather than used directly by analysts.
Things to be aware of
Iceberg needs a catalog to record the current version of each table. The documentation covers several options and the choice affects how engines connect, so it is decided early.
The format also has versions, with newer versions adding capabilities such as row level deletes. Not every engine supports every version at the same pace, so engine support should be checked against the version you plan to use.
First steps
The documentation includes quickstart guides for Spark, Flink and Hive, along with a full specification of the format. The Spark quickstart runs in a container and creates a first Iceberg table with a few SQL statements.
Key features of Apache Iceberg
Capabilities described in the official documentation.
Snapshots of the whole table
Every write produces a new snapshot listing the files in the table, so readers always see a complete and consistent version.
Schema evolution
Columns can be added, dropped, renamed, reordered or have their type widened without rewriting the existing data files.
Hidden partitioning
Iceberg records how partition values are derived from a column, so queries do not have to filter on an extra partition column by hand.
Time travel and rollback
A query can read the table as it stood at an earlier snapshot, and a table can be rolled back to a previous snapshot.
Advantages of Apache Iceberg
Factual advantages that follow from the features above.
Many engines, one table
The format is documented publicly, so different processing engines can read and write the same table rather than each keeping a copy.
Reads never see half a write
A reader works from a committed snapshot, so a job writing new files at the same moment does not change what that reader sees.
Partition layout can change later
Partition evolution lets a table move to a new layout for new data while old data stays where it is.
Statistics let engines skip files
Manifests record the range of values in each data file, so a filtered query opens only the files that could match.
Common use cases for Apache Iceberg
Situations the official documentation describes this tool as being used for.
Shared tables in a data lake
One Iceberg table is written by a Spark job and read by Trino for interactive queries, with both seeing the same state.
Correcting a bad load
A table is rolled back to the snapshot taken before an incorrect batch was written, using the snapshot history Iceberg keeps.
Deleting individual records
Row level deletes remove specific records from a large table without rewriting every file in the partition.
Changing a table schema safely
A column is added or renamed as a metadata operation, leaving the existing data files untouched.
Official website
Everything on this page is based on the official documentation for Apache Iceberg. You can read the source here.
Apache Iceberg official documentation