Disruptor in Data Engineering - Comprehensive Review of Apache Iceberg

Apache Iceberg is an open source table format that has recently gained a lot of attention in the big data world. Managed under Apache Software Foundation, this project has fast become one of the industry favorites to solve some unique and complex big data problems. It is a disruptive and transformative technology which is redefining the landscape of large scale data management. Though this technology belongs in the realm of the data world, it provides unfamiliar capabilities compared to traditional data oriented technologies. As organizations continue to strive for more scalable, cost-effective, performant and reliable ways to handle data, Iceberg sits at the forefront of this to fulfill the expectation. In this paper, we begin by introducing Apache Iceberg and the numerous technical capabilities that make it unique. We discuss its architecture in detail to elucidate how it is designed to accomplish these unique capabilities. We deep dive into how this tool provides flexibility in solving different use cases, its interoperability, the cost savings opportunity it provides and why it has made its place as an attractive tool in the toolbox of organizations. We also discuss about a few other tools which perform similar jobs in this niche technology landscape and compare their features, strengths and weaknesses with Iceberg. We end the discussion with some future research opportunities that may address a few current limitations of Iceberg.