Efficient Compression of XML andJSON Event Log Records

The exponential growth of digital data presents significant storage and transmission challenges. General-purpose compressors often under perform on small input data due to limited context to find and exploit redundancies. This, in turn, prevents the ability to efficiently do individual compression on small inputs like log entries, which is in favour of chunked compression, hindering the ability for random access decompression. Using a design science research methodology, we investigated the characteristics of logs at Ericsson and developed compression strategies for individual XML and JSON log entries that exploit their structural redundancies through shared context. These strategies were benchmarked against commonly used compressors (Gzip, bzip2, LZ4 and Zstandard) using metrics including compression ratio, compression/decompression speed, memory usage and training sensitivity. The results demonstrate that domain-specific compression strategies can significantly improve the compression ratio, but often at the cost of an increase in computation, leading to reduced throughput. Our compression strategies saw compression ratio improvements of 4.8x and 2.8x compared to Gzip, with the latter matching its compression speed. However, results are highly sensitive to training, resulting in decreased compression ratios when context drift occurs. Using a user perception study, we found that industry professionals see potential in adopting a domain-specific compression strategy in certain key areas of their systems. Still, we acknowledged several concerns regarding deployment and infrastructure management, and that a more significant gain would likely be a deciding adoption factor. We conclude that domain-specific compression strategies exploiting structural redundancy and shared context can improve the compression ratio for individual XML and JSON log entries compared to general-purpose ones, offering a viable alternative depending on whether compression ratio or speed is prioritized. Practical applicability hinges on managing context drift and balancing gains for added maintenance complexities.