The smallest eigenvalue of the neural tangent kernel of a deep tanh network in the ordered phase

For a fully connected tanh network of depth $L$ in the infinite-width limit, with weight variance $\sigma_w^2$ and bias variance $\sigma_b^2>0$ in the ordered phase $\chi_1<1$, we determine the smallest eigenvalue of the neural tangent kernel Gram matrix on $n$ distinct inputs of the sphere. It equals $L\chi_1^L\,(q^*+B\,T^*/\chi_1)\,\lambda_{\min}^{\perp}(-A)\,(1+O(1/L))$, where $q^*$ is the fixed-point preactivation variance, $T^*=q^*/(1-\chi_1)$, $B=\chi'(1)$, and $A$ is the matrix of limiting rescaled correlation gaps. That the non-constant eigenvalues are of order $L\chi_1^L$ is due to Xiao, Pennington and Schoenholz; the contribution here is the constant, its positivity, and the extension to inputs of unequal norm. The factor $L$ comes from a double root in the linear recursion for the kernel gap. We prove that $\lambda_{\min}^{\perp}(-A)>0$ for any distinct inputs by showing that every Taylor coefficient of the limiting gap profile is strictly positive, so the rate cannot degenerate further.