Warm-Up Gradient Tracking for Distributed Nonconvex Optimization with Data Heterogeneity

This work focuses on addressing distributed stochastic optimization problems with smooth, nonconvex objective functions over peer-to-peer networks with non-i.i.d. datasets. While existing studies suggest that gradient tracking methods can mitigate the impact of data heterogeneity among nodes, our analysis shows that in nonconvex cases, it can degrade convergence performance owing to the consensus error during the initialization mechanism of gradient tracking, which inherently reflects non-negligible data heterogeneity. To address this issue, we propose an improved warm-up distributed stochastic gradient tracking algorithm, termed W-DSGT, and theoretically demonstrate that reducing the consensus error of the gradient tracking during the initial phase can significantly alleviate the effects of data heterogeneity, thereby leading to enhanced convergence performance.