Erasure Coding Aware Block Placement for Data-Intensive Applications
Erasure Coding (EC) has recently been integrated and deployed in the Hadoop Distributed File System (HDFS) to provide the same fault tolerance guarantees as replication, but with significantly less storage overhead. When EC is used, data reads typically involve only data chunks. In this paper, we study the effect of data chunk distribution on the performance of reads and data-intensive applications, and present the design and evaluation of an erasure coding aware (EC-aware) block placement that balances the distribution of data chunks across nodes. Experimental results show that EC-aware block placement can reduce the execution time of Sort and WordCount applications by up to 25%.
Authors
- Jad Darrous (ORCID: https://orcid.org/0000-0003-4573-4529)
- Shadi Ibrahim (ORCID: https://orcid.org/0000-0002-4306-5280)
Institutions
- Centre National de la Recherche Scientifique (FR)
- Institut de Recherche en Informatique et Systèmes Aléatoires (FR)
- Laboratoire des Sciences du Numérique de Nantes (FR)
- IMT Atlantique (FR)
- Université de Rennes (FR)
- Nantes Université (FR)
Publication Details
- Journal
- ACM SIGOPS Operating Systems Review
- Published
- 2025-08-04
- DOI
- https://doi.org/10.1145/3759441.3759451
- Primary Topic
- Advanced Data Storage Technologies
- Type
- article
- Field-Weighted Citation Impact
- 0.00