DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations

Implicit neural networks including deep equilibrium models have achieved superiortask performance with better parameter efficiency in various applications. However,it is often at the expense of higher computation costs during inference. In thiswork, we identify a phenomenon named heterogeneous convergence that existsin deep equilibrium models and other iterative methods. We observe much fasterconvergence of state activations in certain dimensions therefore indicating thedimensionality of the underlying dynamics of the forward pass is much lower thanthe defined dimension of the states. We thereby propose to exploit heterogeneousconvergence by storing past linear operation results (e.g., fully connected andconvolutional layers) and only propagating the state activation when its changeexceeds a threshold. Thus, for the already converged dimensions, the computationscan be skipped. We verified our findings and reached 84% FLOPs reduction onthe implicit neural representation task, 73% on the Sintel and 76% on the KITTIdatasets for the optical flow estimation task while keeping comparable task accuracywith the models that perform the full update.