Real-time Data Streaming using Kafka, Kinesis, and RabbitMQ

In the present work a comprehensive comparative analysis of the three leading platforms for organizing message streaming — Apache Kafka, Amazon Kinesis and RabbitMQ — is performed with the aim of identifying their architectural features, operational strengths and limitations under conditions of peak loads and stringent latency requirements. The study relies on a comprehensive methodological approach, including a systematic review of current scientific publications, the conduct of comparative performance measurements in laboratory settings and the synthesis of practical case studies of integrating the systems under consideration into real IT landscapes. The obtained results demonstrate that a reasoned choice of platform for stream processing depends on a multitude of interrelated factors: the volume of messages processed, the required throughput metrics and maximum response time, the preferred deployment model (on-premises solution, cloud service or their hybrid), the capabilities for seamless integration with existing services and infrastructure, as well as the project’s budgetary constraints. On the basis of the conducted analysis a unified decision-making methodology is proposed for selecting tools for streaming data processing, adapted to the tasks of data engineers, distributed systems architects and researchers of high-performance information platforms. The material is of practical interest to specialists designing fault-tolerant and scalable distributed message queues, as well as to experts in real-time analytics and cloud solution developers seeking to gain a deeper understanding of the architectural schemes and methods for optimizing throughput applied in Kafka, Kinesis and RabbitMQ. In addition, the research results may be useful to scientists in the field of distributed computing and the Internet of Things, focusing on the theoretical foundations and practical aspects of constructing reliable event-data pipelines.