Contextual Activation Functions for Temporal Systems: A Neuromodulated Approach

We propose Contextual Activation Functions, a family of differentiable activation mechanisms that adapt the nonlinearity of individual hidden units according to an explicit system-state vector and, optionally, a compact summary of recent activity. Inspired by biological neuromodulation, the approach implements a per-unit gate computed from pre-activation, a short history summarizer, and explicit context channels (e.g. energy, environmental signals). The resulting module remains fully differentiable and integrates into recurrent architectures. In a focused evaluation on a synthetic non-stationary temporal forecasting benchmark, the original System-Aware Temporal instantiation achieves a substantial reduction in test mean-squared error relative to a GRU baseline (mean MSE 0.03747 vs. 0.07756 across 10 independent seeds; paired t-test p ≈ 8.5×10⁻¹⁰). We present theoretical analysis of gradient propagation and representational benefits, and we discuss ablations and directions for broader evaluation.