A Highly Scalable Parallel Design for Data Compression

With ever-increasing use of digital data, many applications rely on data compression for their needs related to processing, storage and communication over the network of large volumes of data. While compression saves the memory/disk space and decreases the communication time, there is a considerable runtime spent in this process. Parallel compression algorithms and solutions that are developed to speed up the operations do not scale well on the multi-core CPUs. The data parallel schemes implemented by the prior arts are inefficient in partitioning the data and scaling the performance on the multi-core processors. Another major drawback of the existing multi-threaded compression solutions is the non-compliance to the single-threaded compression format. In this paper, we propose a set of novel and high-performance parallel compression and decompression schemes. We introduce novel designs for dynamic threading based parallel compression and random access point based parallel decompression. With our solution, we mitigate both the scaling issues on multi-core x86 CPUs and format compliance issues encountered in multi-threading the compression operations. Our test results demonstrate massive speedups by manyfolds and performance scaling never seen before on x86 CPUs especially AMD's “ZZn”-based recent processors that come with very high core counts.