ElasticEC: Achieving Fast and Elastic Redundancy Transitioning in Erasure-Coded Clusters

Erasure coding has been extensively deployed in today’s commodity HPC systems against unexpected failures. To adapt to the varying access characteristics and reliability demands, storage clusters have to perform redundancy transitioning via tuning the coding parameters, which unfortunately gives rise to substantial transitioning traffic. We present ElasticEC, a fast and elastic redundancy transitioning approach for erasure-coded clusters. ElasticEC first minimizes the transitioning traffic via proposing a relocation-aware stripe reorganization mechanism and a collecting-and-encoding algorithm. It further heuristically balances the transitioning traffic across nodes. We implement ElasticEC in Hadoop HDFS and conduct extensive experiments on a real-world cloud storage cluster, showing that ElasticEC can reduce 71.1-92.6% of the transitioning traffic and shorten 65.9-90.7% of the transitioning time.