Train Convolutional Neural Networks by Lifted Proximal Operator Method

Currently, the training of deep neural networks primarily relies on gradient-based optimization methods, such as stochastic gradient descent (SGD) and its variants. Although these methods are widely used for model parameter updates, their inherent drawbacks-including gradient vanishing/explosion, susceptibility to local minima, and slow convergence severely limit the performance improvement and training efficiency of deep networks. To overcome the limitations of gradient dependence, non-gradient optimization methods have gradually emerged as a research focus. Among these, the Lifted Proximal Operator Method (LPOM) decomposes complex nonconvex optimization problems into blockwise convex subproblems through variable splitting techniques and achieves parameter updates via alternating iteration strategies, offering a novel approach to circumvent the drawbacks of gradient backpropagation. This study innovatively introduces the LPOM algorithm into the backpropagation process of convolutional neural networks (CNNs), constructing a non-gradient optimization training framework. By decoupling the alternating update mechanism of weight matrices and activation values, the method demonstrates its effectiveness on benchmark datasets such as MNIST and CIFAR-10: it performs comparably to traditional SGD in shallow networks while exhibiting more stable convergence characteristics in deep architectures (e.g., in Tiny ImageNet tasks). Experiments show that this method effectively mitigates gradient anomaly propagation issues, with its optimization trajectory demonstrating significantly superior robustness compared to traditional gradient-driven models like the VGG series, particularly as network depth increases.