Optimizing Inter-Service Communication in Latency-Sensitive Distributed Systems

This research investigates the impact of compression and serialization strategies on latency in message-broker-based distributed systems, focusing on the interplay between these strategies and varying message sizes. Utilizing a custom-designed test communication architecture within a controlled Google Cloud environment, we evaluate various compression utils, including Kafka's built-in mechanisms and external libraries, paired with serialization formats (Avro, Json) for small, medium, and large messages. Our findings demonstrate significant latency reductions compared to uncompressed Json: up to 74% for small messages using Snappy compression with Avro serialization and up to 63% for larger messages using Kafka-native Zstd compression. This research not only addresses a gap in the existing literature regarding message broker-specific optimizations but also provides a foundation for future work exploring the impact of other factors on latency in this widely used distributed messaging platform.