Reusing Polynomial Solvers for Implicit Neural Networks

Preprint; not peer reviewed. Computing gradients in implicit neural networks requires repeated linear solves. A polynomial solver can reuse its coefficients, but changes in the input and network weights make it difficult to ensure that the same polynomial remains accurate. We study this problem for tanh equilibrium layers with explicit weight constraints. A change of variables places all backward systems within one common numerical-range region, so the polynomial can be constructed once and reused with error bounds from existing matrix-function theory. We evaluate the solver in small training tasks and on 108 fixed systems, including 72 systems extracted from trained models. On those 72 systems, at relative residual tolerance 1e-5, the polynomial gives a 2.56–3.16-fold speedup over restarted GMRES when construction cost is spread over 1000 solves, while matching BiCGStab in runtime; the training tasks show similar task metrics across backward solvers. These CPU results support polynomial reuse when matrix products are inexpensive, while leaving its benefit in larger neural-network training to be established.