An Intelligent Optimization-Based Load Balancing Scheduling Algorithm for Flink Tasks

To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink’s default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and load metrics, the scheduling results are dynamically adjusted, resolving the weight deficiency problem of static scheduling strategies. Second, a two-stage optimization approach is adopted: in the Task-to-Slot allocation phase, the Simulated Annealing algorithm is used to optimize resource load balancing among Slots; in the Slot-to-TaskManager allocation phase, the Particle Swarm Optimization algorithm is employed to balance resource differences among TaskManagers. Additionally, a Slots waiting mechanism is introduced, combined with weighted fitness calculations based on feature matrices of tasks, Slots, and TaskManagers, to further optimize global resource allocation. Experiments conducted on the NASA-HTTP-LOG dataset and WordCount job show that for a job with 100 tasks, throughput increases by $9.673 \%$ under globally consistent parallelism and by 41.929% under non-consistent parallelism; for a job with 500 tasks, throughput increases by $10.919 \%$ under globally consistent parallelism and by $16.631 \%$ under non-consistent parallelism. The proposed algorithm effectively resolves resource imbalance and throughput bottlenecks, outperforming both the default scheduling strategy and the Round-Robin static load balancing algorithm.