Phase transitions from linear to nonlinear information processing in neural networks

We investigate a phase transition from linear to nonlinear information processing in echo state networks, a widely used framework in reservoir computing. The network consists of randomly connected recurrent nodes perturbed by a noise and the output is obtained through linear regression on the network states. By varying the standard deviation of the input weights, we systematically control the nonlinearity of the network. For small input standard deviations, the network operates in an approximately linear regime, resulting in limited information processing capacity. However, beyond a critical threshold, the capacity increases rapidly, and this increase becomes sharper as the network size grows. Our results indicate the presence of a discontinuous transition in the limit of infinitely many nodes. This transition is fundamentally different from the conventional order-to-chaos transition in neural networks, which typically leads to a loss of long-term predictability and a decline in the information processing capacity. Furthermore, we establish a scaling law relating the critical nonlinearity to the noise intensity, which implies that the critical nonlinearity vanishes in the absence of noise.