Iceberg Compaction Explained: Bin-Pack vs. Sort

Streaming and frequent batch writes leave Apache Iceberg tables with many small data files. Compaction rewrites them into fewer, larger files so queries read less metadata and open fewer files.

Why small files hurt performance

Every file adds planning overhead and an object-store request. Thousands of small files can make simple queries slow and expensive, even when the total data size is modest.

Compaction strategies

Bin-pack

Bin-pack combines small files into files close to the target size without changing row order. It is the fastest and cheapest strategy and is a good default.

Sort

Sort compaction rewrites data ordered by one or more columns. Queries that filter on those columns can skip more files, but the rewrite costs more compute.

Z-order

Z-order interleaves several columns so filters on any of them benefit. It is useful when queries filter on multiple columns with similar frequency.

[[cta:benchmark]]

How to choose a strategy

Start with bin-pack on a schedule. Move to sort or z-order for large tables where query patterns are stable and file skipping will save more than the extra rewrite cost.

Key takeaways

Compaction is routine maintenance for Iceberg. Schedule it, measure query time and cost before and after, and pick the cheapest strategy that meets your performance goals.

[[cta:demo]]

‍