Skip to main content

05 - Iceberg vs Delta Lake vs Hudi: decision tree

Iceberg, Delta Lake and Hudi compared with executed listings: metadata, schema evolution, maintenance and three questions to choose the table format.

What do you think about this article?

05 - Iceberg vs Delta Lake vs Hudi: decision tree
Data Engineering 2027Photo:AS PhotographyPexels

At a Glance

Iceberg, Delta Lake and Hudi compared with executed listings: metadata, schema evolution, maintenance and three questions to choose the table format.

Iceberg, Delta Lake and Hudi turn a folder of Parquet files into a table with atomic commits, but they do it with different choices, and the right choice depends on three things you can verify: which engines read and write the table, whether data arrive as appends or as updates by key, and who will take care of maintenance. There is no absolute "best", and a serious speed comparison would require a cluster that is not here: I ran Iceberg and Delta Lake live, and I read the Hudi documentation.

Prerequisites. Familiarity with Parquet and with what a table in a data lake is; it helps to have read the article on DuckDB.

What they have in common

The three formats do not replace Parquet: they put it under a metadata structure that says which files make up the table at a given moment, and they advance that structure with an atomic commit. The result is the same for the reader: a consistent snapshot even while someone is writing. The three documentations say it in different words: Iceberg describes optimistic writes that atomically replace the metadata file, the Delta protocol promises snapshot reads and serializable writes, Hudi declares snapshot isolation and optimistic concurrency control among writers.

The star-rating table that circulates in many comparisons is therefore wrong, with "Serializable" for one, "Snapshot" for another and "Read Committed" for Hudi: the Hudi documentation speaks of snapshot isolation and optimistic control, not of Read Committed, and in any case an isolation level is not a quality score. What really changes is where the state lives (log files, manifest tree, timeline), and what you have to do to keep it healthy.

The current versions at the consultation date (2026-11-13), the licenses and who governs the projects are in the table; all three licenses are Apache-2.0 according to the GitHub repositories.

ProjectVersion consultedPython library used hereDeclared governance
Apache Iceberg1.12.0 (2026-09-30)pyiceberg 0.12.0Apache Software Foundation project
Delta Lake4.4.1 (2026-10-05)deltalake 1.6.6 (delta-rs)Linux Foundation sub-project, since 2019
Apache Hudi1.2.1 (2026-09-24)none (written with Spark or Flink)Apache Software Foundation project

The governance datum serves a practical question: who can change the rules of the format? For all three the code is open and a community decides; but a license can change, as the lakeFS case shows in the article lakeFS: Git for data lakes. Reread each project's LICENSE file on the day you choose.

Iceberg: a metadata tree with column ids

Iceberg keeps its state in a JSON metadata file that points to a list of manifests, each of which lists the data files with their statistics. Every column has a numeric identifier that never changes: that is what allows you to rename a column without touching the data. The listing creates a table partitioned by day on a local SQLite catalog (inside the test folder), fills it with two loads, shows time travel and then changes the schema.

import collections, pathlib, shutil
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, LongType, StringType, DoubleType, TimestampType
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform

base = pathlib.Path("lake").resolve()
shutil.rmtree(base, ignore_errors=True); base.mkdir()
catalog = SqlCatalog("local", uri=f"sqlite:///{base}/catalog.db", warehouse=f"file://{base}/warehouse")
catalog.create_namespace("sales")

schema = Schema(NestedField(1, "order_id", LongType(), required=True),
                NestedField(2, "customer", StringType()),
                NestedField(3, "amount", DoubleType()),
                NestedField(4, "ts", TimestampType()))
spec = PartitionSpec(PartitionField(source_id=4, field_id=1000, transform=DayTransform(), name="ts_day"))
table = catalog.create_table("sales.orders", schema=schema, partition_spec=spec)

def batch(start, end, day):
    # order_id is "required" in the table: the Arrow schema must declare it non-null too
    arrow_schema = pa.schema([pa.field("order_id", pa.int64(), nullable=False), pa.field("customer", pa.string()),
                              pa.field("amount", pa.float64()), pa.field("ts", pa.timestamp("us"))])
    return pa.table({"order_id": list(range(start, end)), "customer": [f"c{i % 7}" for i in range(start, end)],
                     "amount": [float(i % 50) for i in range(start, end)], "ts": [day] * (end - start)}, schema=arrow_schema)
import datetime as dt
table.append(batch(0, 1000, dt.datetime(2026, 11, 1, 10)))
table.append(batch(1000, 2000, dt.datetime(2026, 11, 2, 10)))
first = table.history()[0].snapshot_id
print("snapshots:", len(table.snapshots()), "| current rows:", table.scan().to_arrow().num_rows)
print("time travel, first snapshot:", table.scan(snapshot_id=first).to_arrow().num_rows, "rows")

# schema evolution: metadata only, the data files stay the same
files_before = sorted(p.name for p in base.rglob("*.parquet"))
with table.update_schema() as upd:
    upd.add_column("channel", StringType())
    upd.rename_column("customer", "customer_id")
table = catalog.load_table("sales.orders")
files_after = sorted(p.name for p in base.rglob("*.parquet"))
print("Parquet files identical before and after the evolution:", files_before == files_after, f"({len(files_after)} files)")
print("columns:", [f.name for f in table.schema().fields])
read = table.scan().to_arrow()
print("channel in the old rows:", set(read["channel"].to_pylist()), "| column ids:", [f.field_id for f in table.schema().fields])

# what is on disk
kinds = collections.Counter(("metadata.json" if p.name.endswith("metadata.json") else p.suffix) for p in base.rglob("*") if p.is_file() and "catalog" not in p.name)
print("files on disk by type:", dict(sorted(kinds.items())))

# Output:
# snapshots: 2 | current rows: 2000
# time travel, first snapshot: 1000 rows
# Parquet files identical before and after the evolution: True (2 files)
# columns: ['order_id', 'customer_id', 'amount', 'ts', 'channel']
# channel in the old rows: {None} | column ids: [1, 2, 3, 4, 5]
# files on disk by type: {'.avro': 4, '.parquet': 2, 'metadata.json': 4}

Three things to read in the output. After two append calls there are two snapshots, and the scan of the first returns only the first 1000 rows: that is time travel, which costs an identifier and no copies. The schema evolution (adding channel, renaming customer to customer_id) did not rewrite any Parquet file: the two files are the same before and after, and the column ids 1-5 did not change. On disk there are 4 JSON metadata files (one per commit), 4 Avro files (manifest lists and manifests) and the 2 data Parquet files.

Two pitfalls that the run showed. The order_id column is declared required and the Arrow schema must declare it non-null, otherwise append fails with a field comparison; and partitioning by day requires the pyiceberg-core package, without which the error is NotInstalledError. Finally, maintenance is not automatic: the Iceberg documentation recommends expiring snapshots with expireSnapshots and compacting files with the rewriteDataFiles action, both operations to be scheduled.

Delta Lake: a transaction log

Delta keeps its state in a _delta_log folder of numbered files, each with the actions of one commit: add and remove of files, metaData, protocol. To rewind the table the log is reread up to the desired version; periodically a Parquet checkpoint file summarizes the state so that not everything has to be reread. The listing uses the deltalake library without Spark, and goes through the log, time travel, schema evolution and maintenance.

import json, pathlib, shutil
import pyarrow as pa
from deltalake import DeltaTable, write_deltalake

path = pathlib.Path("delta_orders").resolve()
shutil.rmtree(path, ignore_errors=True)

def batch(start, end):
    return pa.table({"order_id": list(range(start, end)), "customer": [f"c{i % 7}" for i in range(start, end)],
                     "amount": [float(i % 50) for i in range(start, end)]})

write_deltalake(str(path), batch(0, 1000))                       # version 0
write_deltalake(str(path), batch(1000, 2000), mode="append")    # version 1
log = sorted(p.name for p in (path / "_delta_log").iterdir())
print("log files:", log)

actions = [list(json.loads(line))[0] for line in (path / "_delta_log" / log[0]).read_text().splitlines()]
print("actions in version 0:", actions)

dt = DeltaTable(str(path))
print("current version:", dt.version(), "| rows:", dt.to_pyarrow_table().num_rows)
print("time travel, version 0:", DeltaTable(str(path), version=0).to_pyarrow_table().num_rows, "rows")

# schema evolution: a new column requires saying so explicitly
new = batch(2000, 2100).append_column("channel", pa.array(["web"] * 100))
try:
    write_deltalake(str(path), new, mode="append")
except Exception as e:
    print("without schema_mode:", type(e).__name__)
write_deltalake(str(path), new, mode="append", schema_mode="merge")
dt = DeltaTable(str(path))
print("columns after the schema merge:", [f.name for f in dt.schema().fields])
print("channel in the old rows:", sorted({str(v) for v in dt.to_pyarrow_table()["channel"].to_pylist()}))

# maintenance: it is an explicit operation, not an automatic one
dt.delete("order_id < 500")
dt = DeltaTable(str(path))
print("files in the active log before compact:", len(dt.file_uris()))
dt.optimize.compact()
print("files after compact:", len(DeltaTable(str(path)).file_uris()), "| rows:", DeltaTable(str(path)).to_pyarrow_table().num_rows)
print("dry-run vacuum, files that are candidates for removal:", len(DeltaTable(str(path)).vacuum(retention_hours=0, dry_run=True, enforce_retention_duration=False)))

# checkpoint: a Parquet summary of the state, written on request (here) or automatically after N commits
dt.create_checkpoint()
print("after the checkpoint:", sorted(p.name for p in (path / "_delta_log").iterdir() if "checkpoint" in p.name or p.name == "_last_checkpoint"))

# Output:
# log files: ['00000000000000000000.json', '00000000000000000001.json']
# actions in version 0: ['commitInfo', 'protocol', 'metaData', 'add']
# current version: 1 | rows: 2000
# time travel, version 0: 1000 rows
# without schema_mode: SchemaMismatchError
# columns after the schema merge: ['order_id', 'customer', 'amount', 'channel']
# channel in the old rows: ['None', 'web']
# files in the active log before compact: 3
# files after compact: 1 | rows: 1600
# dry-run vacuum, files that are candidates for removal: 4
# after the checkpoint: ['00000000000000000004.checkpoint.parquet', '_last_checkpoint']

The log after two writes has two files, and the first contains the actions commitInfo, protocol, metaData and add. Adding a column without saying so is rejected with SchemaMismatchError: you need schema_mode="merge", and the old rows read None in the new column. After a deletion, the number of active files was 3 and compaction brought them to 1 with 1600 rows, while the dry-run vacuum lists the 4 files no longer referenced. In this listing I forced it with retention at zero hours only because it is a dry run on test data: the Delta documentation gives 7 days as the default retention for deleted files, and on a real table it must not be reduced lightly.

The comparison with Iceberg, experiment for experiment: the same schema evolution but with an explicit action; the same time travel but by version number; the same manual maintenance (optimize.compact, vacuum, create_checkpoint). Neither is "automatic": whoever says so confuses the availability of a command with its execution.

Hudi: the timeline and the record keys

Hudi was born for incremental ingestion and for updates by key, and it shows in the model: every record has a key, every table a timeline of actions, and an ordering field decides who wins between two versions of the same record. There are two table types. In Copy-on-Write every update rewrites the base files, so writing costs more and reading is simpler; in Merge-on-Read updates are appended to log files that the read applies, so writing is faster and reading costs more, as the documentation's comparison table says.

# Schema to adapt, not executed: Hudi is written with Spark (or Flink) and there is no Spark here.
# Options and package are those of the Hudi 1.2.1 quick start guide, not verified on a cluster.
#   pyspark --packages org.apache.hudi:hudi-spark3.5-bundle_2.13:1.2.1 \
#           --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("hudi-orders").getOrCreate()
options = {
    "hoodie.table.name": "orders",
    "hoodie.datasource.write.recordkey.field": "order_id",        # what identifies the same record
    "hoodie.datasource.write.partitionpath.field": "day",
    "hoodie.table.ordering.fields": "ts",                         # between two versions of the same record the higher ts wins
    "hoodie.datasource.write.operation": "upsert",
    "hoodie.datasource.write.table.type": "MERGE_ON_READ",         # or COPY_ON_WRITE
}
changes = spark.read.json("cdc/orders.json")                      # new and changed records
changes.write.format("hudi").options(**options).mode("append").save("lake/orders_hudi")

This schema was not executed: Hudi is written with Spark or Flink, and I started neither. The options are those of the version 1.2.1 quick start guide. A difference from old tutorials: the guide uses hoodie.table.ordering.fields for the ordering field, while many examples report the earlier name precombine.field. Check the documentation for your version before copying the options. Concurrency is documented as snapshot isolation among writers, readers and table services, with optimistic control among writers.

Reading the three tables from DuckDB

To read these formats without Spark, DuckDB has two official extensions, delta and iceberg, which install themselves on first use. The listing reads the two tables created above and reports the number of rows and the sum of the amount. The functions are called delta_scan and iceberg_scan (not read_delta, which appears in some tutorials). Iceberg wants the path of the most recent metadata file.

import duckdb, pathlib

con = duckdb.connect()
con.execute("INSTALL delta; LOAD delta; INSTALL iceberg; LOAD iceberg;")

# Delta: delta_scan on the table folder
print("delta  :", con.execute("SELECT count(*), sum(amount) FROM delta_scan('delta_orders')").fetchall())

# Iceberg: iceberg_scan wants the most recent metadata file
metadata = sorted(pathlib.Path("lake").rglob("*.metadata.json"))[-1]
print("iceberg:", con.execute(f"SELECT count(*), sum(amount) FROM iceberg_scan('{metadata}')").fetchall())

# Output:
# delta  : [(1600, 39200.0)]
# iceberg: [(2000, 49000.0)]

The counts match what was written: 1600 rows in the Delta table after the deletion, 2000 in the Iceberg one. From the DuckDB 2.0 documentation: Iceberg tables read directly from the metadata are read-only, while writing requires a REST catalog; the Delta extension reads and can create tables and add rows. I did not find Hudi among the core extensions listed in the documentation, so for Hudi you need another engine. The same documentation also mentions DuckLake, a format of the DuckDB project, with version 1.1 published together with DuckDB 2.0. To then process the results with a DataFrame, see Polars and pandas and the pipeline with DuckDB and Polars.

How to choose

The choice reads better as three questions in sequence, because the first narrows the field for the others. First: which engines must read and write? If you have Spark and Flink and want updates by key from a stream of changes, Hudi is built for that and its documentation puts it at the center; if you want tables read by many different engines, Iceberg has an engine-independent specification, now at version 3 with new data types; if you already work with the Delta ecosystem, the protocol is documented and the Rust library lets you write without Spark.

Second: how does the data arrive? For batch loads in append mode, all three are fine and the difference shifts to maintenance. For a stream of updates by key, Hudi's model (record key and ordering field) is the one its documentation puts most explicitly at the center, while Iceberg and Delta cover it with merge and row-level delete operations. Third: who runs the maintenance? For all three these are operations to schedule (snapshot expiry, compaction, cleanup of orphan files, checkpoints), and the organizational cost of those operations counts as much as performance.

The table summarizes where to look for each criterion and how to verify it on your own case, without assigning scores:

CriterionWhere to lookHow to verify
Engines that readDocumentation of your engine (DuckDB, Trino, Spark, Flink)List of the extensions and connectors of the version you use
Engines that writeSame; for Iceberg from DuckDB you need a REST catalogTry a write on a copy of the data
Updates by keyHudi documentation (record key, ordering field) and the merge operations of Iceberg and DeltaA test load with duplicate and out-of-order records
Schema evolutionIceberg: column ids; Delta: schema_modeAdd and rename a column as in the listings and check the files
MaintenanceMaintenance pages of each projectSchedule and try snapshot expiry, compaction, cleanup

If after the three questions several formats remain admissible, the winner is the one your team can keep running at night: in data, the worst format is the one nobody knows how to repair. To put one of these formats inside a complete architecture, go to the last article of the series, Modern Data Stack: a minimal architecture; to read them with a lightweight engine, DuckDB in production.

What you can do now

You can explain what the three formats have in common and where they diverge, create and query an Iceberg table and a Delta one without Spark, see with your own eyes that schema evolution does not rewrite the data, read them from DuckDB with the right functions, and choose with three questions instead of a star ranking. You also know that no speed figure in this article was invented: there are none, and those in the comparisons you find online should be accepted only if they declare machine, data, versions and method.

Frequently asked questions

Which format is the fastest?

I did not measure it: a fair comparison requires the same engine, the same data and maintenance at the same stage, and there is no cluster here. Distrust tables without machine, versions and method: the article reports none.

Do you really need Spark or a cluster to use them?

For Iceberg and Delta, no: the listings above create, write and read the tables with Python on a single computer. For Hudi the quick start guide uses Spark; the documentation also mentions hudi-rs, a native Rust implementation with Python bindings, which I did not try.

Can I read the same table with several engines?

Yes, if each engine supports the format: in the experiment DuckDB read the table written by pyiceberg and the one written by deltalake. Always check which format versions and which features (deletions, new types) the engine can read.

Does a format replace lakeFS?

No: the formats version a table, lakeFS versions a set of objects with branches and merges. They can coexist; the details and the license are in the dedicated article.

Sources

Versions and licenses come from the releases and the GitHub repositories of the three projects, consulted on 2026-11-13 through their respective public addresses. The Iceberg, Delta and DuckDB listings were run with Python 3.14.4, pyiceberg 0.12.0, deltalake 1.6.6 and DuckDB 2.0.0; the first two and the last twice with identical output. The Hudi schema was not executed.

Try These Tools

Free developer tools related to this topic.

Commenti

Caricamento commenti...

Sign in to comment

New articles and tools on Telegram too

The Telegram channel flags the site's new articles and tools.

Did you like this article?

Explore other content on the blog or discover my projects