AMC: Adaptive Learning Rate Adjustment Based on Model Complexity

An optimizer plays a decisive role in the efficiency and effectiveness of model training in deep learning. Although Adam and its variants are widely used, the impact of model complexity on training is not considered, which leads to instability or slow convergence when a complex model is trained. To address this issue, we propose an AMC (Adam with Model Complexity) optimizer, which dynamically adjusts the learning rate by incorporating model complexity, thereby improving training stability and convergence speed. AMC uses the Frobenius norm of the model to measure its complexity, automatically decreasing the learning rate of complex models and increasing the learning rate of simple models, thus optimizing the training process. We provide a theoretical analysis to demonstrate the relationship between model complexity and learning rate, as well as the convergence and convergence bounds of AMC. Experiments on multiple benchmark datasets show that, compared to several widely used optimizers, AMC exhibits better stability and faster convergence, especially in the training of complex models.