Fault Injection and Verification for Lakehouse Table Maintenance: Crash-Testing Apache Iceberg Compaction, Snapshot Expiry and Orphan Removal

Apache Iceberg tables need regular maintenance: small-file compaction (rewrite_data_files), snapshot expiry (expire_snapshots) and orphan-file removal (remove_orphan_files). These operations delete or replace files that readers and writers depend on, while ingestion continues. I built a fault-injection (chaos engineering) checker that runs each operation against a live ingest, kills or pauses maintenance at chosen phases, then checks the table against a ledger of acknowledged writes. The checker scores three classes separately: A (data integrity), B (retained snapshots and tags), and C (whether a maintenance call's reported progress matches what committed). An independent reader (pyiceberg) repeats the A and B checks on every cell. I ran 123 cells on three builds (1.7.1, 1.12.0, and main at commit 2d89aa4): 369 cells in total. Classes A and B recorded no violation in any of the 369 cells; the two readers never disagreed. I found no novel violation. The only class C flags map to an existing, open issue (#17202): a partial-progress rewrite returns success while only some commits land after a concurrent expiry removes its starting snapshot. The signature appeared in 8 of 8 in-matrix race cells on 1.7.1. A gated re-run forcing real overlap between the expiry commit and the rewrite's commits flagged 3 of 3 cells on 1.7.1, 8 of 8 on 1.12.0 and 7 of 8 on main. Reported counts were exact in every flagged cell; only failed_data_files_count = 0 misleads. A positive control for live-file deletion (#17304, closed as spec-discouraged and contested) is detected by classes A and B on all three builds through the core API, but not through the Spark procedure or on a tagged table. The negative result is narrow: it covers an unpartitioned, append-only, tagged table, a Postgres JDBC catalog and single-node MinIO, and cannot speak to the phases my rig never reached.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24412832
Primary Topic
Advanced Database Systems and Queries
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Fault Injection and Verification for Lakehouse Table Maintenance: Crash-Testing Apache Iceberg Compaction, Snapshot Expiry and Orphan Removal

Sajish Krishnan
Zenodo (CERN European Organization for Nuclear Research)
Advanced Database Systems and Queries
preprint

Fault Injection and Verification for Lakehouse Table Maintenance: Crash-Testing Apache Iceberg Compaction, Snapshot Expiry and Orphan Removal

Sajish Krishnan
preprint en

Abstract

Apache Iceberg tables need regular maintenance: small-file compaction (rewrite_data_files), snapshot expiry (expire_snapshots) and orphan-file removal (remove_orphan_files). These operations delete or replace files that readers and writers depend on, while ingestion continues. I built a fault-injection (chaos engineering) checker that runs each operation against a live ingest, kills or pauses maintenance at chosen phases, then checks the table against a ledger of acknowledged writes. The checker scores three classes separately: A (data integrity), B (retained snapshots and tags), and C (whether a maintenance call's reported progress matches what committed). An independent reader (pyiceberg) repeats the A and B checks on every cell. I ran 123 cells on three builds (1.7.1, 1.12.0, and main at commit 2d89aa4): 369 cells in total. Classes A and B recorded no violation in any of the 369 cells; the two readers never disagreed. I found no novel violation. The only class C flags map to an existing, open issue (#17202): a partial-progress rewrite returns success while only some commits land after a concurrent expiry removes its starting snapshot. The signature appeared in 8 of 8 in-matrix race cells on 1.7.1. A gated re-run forcing real overlap between the expiry commit and the rewrite's commits flagged 3 of 3 cells on 1.7.1, 8 of 8 on 1.12.0 and 7 of 8 on main. Reported counts were exact in every flagged cell; only failed_data_files_count = 0 misleads. A positive control for live-file deletion (#17304, closed as spec-discouraged and contested) is detected by classes A and B on all three builds through the core API, but not through the Spark procedure or on a tagged table. The negative result is narrow: it covers an unpartitioned, append-only, tagged table, a Postgres JDBC catalog and single-node MinIO, and cannot speak to the phases my rig never reached.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Fault Injection and Verification for Lakehouse Table Maintenance: Crash-Testing Apache Iceberg Compaction, Snapshot Expiry and Orphan Removal — Sajish Krishnan · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS