Making Room for Domain-Specific Algorithms in Big Data Compression Pipelines

The exponential growth of data in astrophysics, seismology, and climate modeling demands efficient compression strategies to optimize storage, transmission, and computational resources. This paper presents a modular and extensible compression framework tailored for big data applications, enabling dynamic reconfiguration of compression pipelines based on contextual requirements. The proposed framework allows for the integration of domain-specific algorithms aiming at achieving greater compression ratios by selecting compression strategies that are more suitable for a given dataset. The proposed framework was evaluated through a comprehensive benchmarking process and compared with two traditional data compression tools, FPZIP and ZFP, regarding both the compression ratio and computational performance. The results highlight the advantages of adaptive, fine-tuned compression strategies in managing large-scale datasets efficiently, as the proposed approach achieves greater compression ratios due to its flexibility in selecting compression strategies appropriate for a given domain.