ASSG: Enhanced Workload Balancing via Adaptive State Scheduling Granularity Approach for Stateful Distributed Stream Processing

The dynamic changes in the distribution of data attributes within the data stream lead to unbalanced workloads in stream processing operators, often necessitating state migration in these operators. However, significant time overhead associated with state migration results in latency peaks and interruptions in stream applications. Ignoring the impact of state scheduling granularity exacerbates the time overhead during migration. Focusing on state migration efficiency, this article presents the Adaptive State Scheduling Granularity (ASSG) approach for stateful distributed stream processing. The ASSG conducts dynamic workload balancing at the granularity of state subgroups, where the number and size of any subgroup can be adaptively adjusted to reduce the time of collecting workloads and generating balancing plans. Moreover, since the idle instances may experience higher workloads than average due to receiving the migrated states, we propose a parallel migration strategy. They are enabled to post low states to other instances while receiving the high ones, alleviating the workload imbalance without extending migration duration. ASSG has been fully implemented on Apache Flink, which demonstrates a 23.84% reduction in average latency compared to state-of-the-art workload balancing schemes and achieves at least a 10.91% reduction in the time required for state migration through adaptive granularity adjustment.