Real-time Apache Kafka Server Resource (Data Disk) Optimization by A Load-shedding ML/AI Engine for Consumer-Driven Retention in Context of IoT as Consumers.

Apache Kafka is a high-throughput and distributed messaging system designed to handle large volumes of data feeds with scalability and reliability, serving as an essential part of event-driven architecture. Despite its advantages and popularity, Kafka’s high degree of flexibility reduces the incentive for academic researchers to develop universal resource utilization strategies, making the study of this kind underexplored. A few of very recent studies have initiated research to address this generalizability issue, placing emphasis on responding backpressure from producer-side, but with limited consideration given towards consumer-side. In many IoT scenarios, though, a messaging middleware follows a fire-and-forget model, where message durations are transient, ending once the message processed by its recipient. This study seeks to address this research gap in developing generalized, real-time resource allocation strategies tailored to any Kafka-involved architecture characterized by such IoT cases with minimal storage requirements, by allowing it to learn operational pattern of the consumer-side. This project is initialized with an artificial IoT use case to produce synthetic datasets, ensuring a controlled and repeatable environment with experiments. The result of this study, based on quantitative analysis, demonstrates that a single broker maintains linearity of message lifetime against producing time on a per-batch basis, which reflects Kafka’s append-only design principle as the chosen data structure. Aligning well with this fundamental, the project firstly demonstrates the basis for future implementation of resource optimization with a potential to generalize. This knowledge basis obtained is subsequently applied, upon which a persistent data storage load-shedding engine is developed. Working together, the engine modules coordinate consumers behavior data, make predictions for log retention time setting for topics, and then automate configuration adjustments in real-time for saving persistent storage. Overall, this project serves as a proof of concept for Kafka-based architecture in IoT use cases sharing specific characteristics, with a light-weight local setup. Even though it relies on an artificial use case with synthetic data generated, it fills a gap in high-level Apache Kafka performance and resource utilization research with a possibility of broad-grained optimization around broker layer and creates opportunities to prevent disk failure using unsupervised machine learning algorithms.