A High-Level API for End-to-End Data Compression in Multi-GPU Cluster Applications

Abstract The increasing complexity of high-performance computing (HPC) systems, particularly those utilizing a multi-GPU cluster architecture, presents significant challenges for developers. Low-level, vendor-specific application programming interfaces (APIs) often require deep technical knowledge, complicating maintenance and reducing performance portability. This complexity is exacerbated when large data sizes or bandwidth requirements necessitate data compression. Integrating such end-to-end compression into existing codebases dealing with distributed memory multi-GPU clusters is often labor-intensive and error-prone, particularly when flexibility in the choice of compression algorithm and implementation is desired. In this work, we propose a high-level, user-friendly end-to-end data compression API integrated into the Celerity runtime system, built on the SYCL programming model. By abstracting the complexities of integrating various compression algorithms, we aim to enhance data transfer and storage efficiency, without a large cost in development complexity. Our API supports multiple compression types and memory layouts, enabling developers to leverage compression without extensive modifications to their existing codebases. We evaluate this API and its prototype implementation in various benchmarks, demonstrating its effectiveness in reducing storage and bandwidth requirements, while maintaining high performance.