DMGA: A Directional-Mutation Genetic Algorithm for CDMS Problem of Kafka

With the increasing role of distributed message systems (DMS) represented by Kafka in handling large-scale data streams, it is critical to improve system performance by optimizing configuration parameters in different production environments. Due to the target system possessing numerous configuration parameters and the complex interactions between them, parameter optimization becomes a complex and time-consuming process that relies heavily on human effort and an iterative trial-and-error approach. At present, this type of configuration parameter problem has been classified as the Configuration of Distributed Message Systems (CDMS) problem, for which the goal is to obtain the optimal set of parameters for achieving the best performance. In order to deal with this complex problem, an automatic optimization method is proposed in this paper, which combines an automatic script-based deployment and running framework with an improved Genetic Algorithm (GA) in cooperation with the Random Forest (RF) and statistical methods, named the Directional-Mutation Genetic Algorithm (DMGA). The framework and algorithm are designed to find the best configuration efficiently, obtaining optimal parameters accurately without human supervision. Actual results of comparison experiments based on the Kafka system show that the DMGA algorithm significantly improves the data throughput performance of Kafka, and its optimization speed is more efficient than the basic GA, highlighting its potential to quickly optimize Kafka's operational efficiency in different scenarios.