A Novel Approach to Gradient Evaluation and Efficient Deep Learning: A Hybrid Method

Deep learning faces significant challenges in efficiently training large-scale models. These issues are closely linked, as efficient training often depends on precise and computationally feasible gradient calculations. This work introduces innovative methodologies to improve deep learning network (DLN) training in complex systems. A novel approach to DLN training is proposed by adapting the block coordinate descent (BCD) method, which optimizes individual layers sequentially. This is combined with traditional batch-based training to create a hybrid method that harnesses the strengths of both techniques. Additionally, the study explores Iterated Control Random Search (ICRS) for initializing parameters and applies quasi-Newton methods like L-BFGS with restricted iterations to enhance optimization. By tackling DLN training efficiency, this contribution offers a comprehensive framework to address key challenges in modern machine learning. The proposed methods improve scalability and effectiveness, especially for handling complex real-world problems. Examples from Process Systems Engineering illustrate how these advancements can directly enhance the training of large-scale systems.