Evolutionary Computation: Stateless Population Flow Design for Cloud Systems

Evolutionary computation offers powerful optimization capabilities, but deploying population-based algorithms in cloud environments presents challenges related to state management, scalability, communication overhead, and fault tolerance. This paper presents KafkaEO, a cloud-native framework that organizes evolutionary computation through stateless services and event-driven population flows. The proposed architecture decomposes populations into micro-populations processed by independent evolutionary operators, using Apache Kafka for asynchronous communication and a lightweight controller for experiment coordination. The framework was implemented using Python, DEAP, IBM Cloud Functions, and Apache Kafka. Its performance was evaluated on selected BBOB benchmark functions across problem dimensions ranging from 2 to 40, with comparisons against the EvoSpace platform over 15 independent runs. Results indicate comparable convergence behavior, although KafkaEO incurred approximately 5–8% additional function evaluations on average. Scalability experiments involving 2–16 workers demonstrated reduced execution time, while total benchmark experiment costs remained below $15. A convolutional neural network hyperparameter optimization case study on MNIST achieved 98.7% validation accuracy and 98.5% test accuracy. These findings demonstrate the feasibility of primarily stateless, event-driven evolutionary computation for moderate-scale cloud optimization, while highlighting opportunities for improving efficiency and scaling.