PLZW: FPGA-based Pipelined LZW for lossless compression of edge data

Abstract The rapid growth of IoT devices and data in edge computing applications has created a pressing need for efficient lossless compression of end-edge data. The Lempel-Ziv-Welch (LZW) algorithm, widely used in these scenarios, faces bottlenecks in compression speed due to its inherent data dependency. Unlike existing multi-cycle architectures, we propose PLZW, a pipeline-based LZW acceleration architecture with three novel mechanisms: a data bypass mechanism that allows data to pass directly between pipeline stages, minimizing stalls; a multi-level caching mechanism to reduce pipeline stalls caused by long dictionary update times; and a hash prediction mechanism to improve dictionary lookup hit rates and reduce stall frequency. Our field-programmable gate array (FPGA) implementation demonstrates a 2.5 $$\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mo>×</mml:mo> </mml:math> speedup over existing state-of-the-art methods while maintaining competitive power consumption, resource utilization, and compression ratio. Furthermore, the resource utilization remains below 1% of total FPGA resources, and the proposed mechanisms improve the dictionary first-hit rate by 35.8%. This work offers a high-speed, lightweight, generic solution for lossless compression of end-edge data. PLZW provides a reference implementation for optimizing hardware-based data compression in real-time and resource-constrained applications.