S1 Appendix -

<p>A. Experimental details. <b>Fig A. A-B</b> Task independent stability metrics in activation parameter space. <b>C-D</b> Test accuracy in activation parameter space for the psMNIST task under two different learning scenarios. B. Performance: supplemental B.1 Comparison between homogeneous and heterogeneous activation functions. <b>Fig B</b>. Comparison between homogeneous and heterogeneous activation functions. Labels and perturbation details follow <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1012567#pcbi.1012567.g002" target="_blank">Fig 2</a>. B.2 Further details on learning differences and performance in the static setting. B.3 Learned adaptation offers transfer learning advantages. <b>Fig C</b>. Trajectories of the activation parameters during retraining on the modified MNIST images. C. Adaptation: supplemental. C.1 Fractional differentiation. Fig D. (<b>Top</b>) Graph of a step to linear-increase function (right), then fractional order (<i>α</i> = 0.15) differentiated (left). (<b>Bottom</b>) Saturation <i>s</i><sub><i>t</i></sub> as a function of the time (left), for varying external drives <i>ξ</i> ∈ [0, 30] with the usual range applied during a stimulation period framed by the two dashed green lines. See next Fig E for colorbar. (right) The saturation signals <i>s</i><sub><i>t</i></sub> fractionally integrated with <i>α</i> = 0.15 reveal step to linear increase signals during the stimulation period. <b>Fig E</b>. Task: psMNIST. Random seed #: 400. Colorbar applies to whole figure. (<b>top-right</b>) mean ARU hidden-states for non-interacting ARUs, just as main text’s setting. For other panels, see respective titles. <b>Fig F</b>. Task: psMNIST. Random seed #: 500. Fig G. Task: sCIFAR10. Random seed #: 403. <b>Fig H</b>.<b>(a, c)</b> Distribution, over neurons, of fractional order <i>α</i> for interacting ARUs (<b>a</b>) and non-interacting, “Isolated”, ARUs (<b>c</b>). The order is established by minimizing the MSE between the fractional order differentiated signal of ARU activity and the step drive applied (<i>ξ</i> = 20 during <i>t</i> ∈ [100, 200)). If we apply the same procedure to the mean network activity, after averaging over neurons, we obtain the single estimate “<i>α</i> of mean resp” indicated by the black line. We report a Gaussian Kernel Density Estimate (KDE) of the distribution, with “scott” bandwidth estimation procedure. <b>(b, d)</b> Average activity for ARUs with the indicated fractional order <i>α</i>, for interacting ARUs (<b>b</b>) and non-interacting, “Isolated”, ARUs (<b>d</b>). C.2. Dynamic regularization. C.3. Sensitivity analysis for the perturbations experiments. <b>Fig I</b>. Sensitivity analysis for the step drive experiment. Lower is better. ARUN performs the best. <b>Fig J</b>. Sensitivity analysis for the Sinusoidal transformation on inputs, varying phase and amplitude alternatively. Lower is better. ARUN performs the best on average. <b>Fig K</b>. Sensitivity analysis for the noisy step drive experiment for the sCIFAR10 task. Higher is better. <b>Fig L</b>. Sensitivity analysis for the sinusoidal transformation on inputs, varying the frequency, for the sCIFAR10 task. Higher is better. C.4. Testing the evolutionary plausibility of our adaptive units. <b>Fig M</b>. Performance on the psMNIST classification task and robustness to the noise perturbed, step drive transformed and sine transformed inputs. The mean and standard deviation across three different initializations are shown. C.5. Gradient contribution according to position in input sequence. <b>Fig N</b>. Frobenius norm of the hidden-to-hidden weight matrix <i>W</i><sub><i>hh</i></sub> gradient contribution of a given input element, or pixel, as a function of that element’s position in the input sequence. The sequences are of length 784 and elements closer to position 0 are closer to the beginning of the input sequences. The gradients are computed in trained RNN+ReLU, RNN+<i>γ</i> heterogeneous and ARU networks on the psMNIST training set. Mean and standard deviation across three random initialization are shown. D. A primer on Lyapunov exponents. D.1. Definition of Lyapunov exponents. D.2. Oseledets theorem. D.3. The QR algorithm. D.4. Link to RNNs.</p> <p>(PDF)</p>