Fractional Gradient Descent With Matrix Stepsizes for Non-Convex Optimization

Fractional derivatives generalize integer-order derivatives, making them relevant for studying their convergence in descent-based optimization algorithms. However, existing convergence analysis of fractional gradient descent (FGD) is limited in both methods and settings. This article bridges these gaps by establishing convergence guarantees for FGD on a broader class of non-convex functions, known as matrix-smooth functions. We leverage the matrix smoothness properties of the function to prove convergence and accelerate the FGD iterates. We propose two novel stochastic fractional descent algorithms, named compressed FGD (CFGD), incorporating a matrix-valued stepsize to minimize matrix-smooth non-convex objectives. Our theoretical analysis covers both single-node and distributed settings and shows that matrix stepsizes better capture the structure of the objective, leading to faster convergence than scalar stepsizes. In addition, we highlight the importance of matrix stepsizes to leverage model structure effectively. To the best of our knowledge, this is the first work to introduce FGD in a federated/distributed setting.