Research on the equivalence principle and module design of erasure code based on multitenants

Erasure coding technology can reduce costs, improve data reliability, and enhance the usefulness of the system. This technology has been widely designed and used in distributed storage systems. However, in the case of multi-tenancy, none of the popular distributed storage systems support user-level flexible and configurable encoding modules. To solve the problem that the cost and reliability cannot be controlled by custom encoding parameters, the proof of equivalence principle under different code construction schemes is provided, and then the erasure encoding module under multi-tenancy is designed. This module allows a flexible selection of stripe ratio parameters and code algorithms, and data conversion under different algorithms. Based on the generator polynomial, the code library is provided which is optimized by avx2 instruction sets. Compared with the open-source generator matrix-based code solution, the library can resist silent errors. Finally, the analysis and evaluation of the throughput performance of the three code libraries that support avx2 instruction sets is provided. The experiment results show that ISA-L library could maintain high encoding throughout with the growth of the size of the stripe. The design of the flexible erasure code module under multi-tenancy environment has filled the gap of the distributed storage system in the refined control of cost and reliability.