Efficient Deep Equilibrium Models: Denoising Regularization and Average Fixed-Point Initialization to Reduce Function Evaluations

Deep Equilibrium models efficiently implement the convergence of weight-shared layers to a fixed-point. However, during training, the number of function evaluations increases significantly due to the increasing complexity of the model, leading to long inference times. As a countermeasure, regularization of the Jacobian using Hutchinson’s estimator has been proposed and lead to an overall decrease in the number of function iterations during inference. We propose to use denoising regularization as an alternative approach for Jacobian regularization. This approach decreases the number of function evaluations further. Moreover, we further decrease the number of function evaluations by using the average fixed-point as a starting point instead of an arbitrary fixed value whilst keeping the performance on a similar level.