Contextual Activation Functions for Temporal Systems: A Neuromodulated Approach

This work introduces Contextual Activation Functions (CAF), a theoretical framework for state-dependent nonlinear transformations in neural computation1. Drawing inspiration from neuromodulatory mechanisms in biological systems, CAF generalizes standard activation functions by conditioning the nonlinearity on an explicit context vector that encodes system state or temporal dynamics. This enables a structured form of input-dependent nonlinearity that can adapt effective gain, saturation regimes, and information flow based on system state, making it ideal for modeling non-stationary dynamics The framework is formulated as a differentiable, multiplicative gating mechanism that modulates a base activation function. Given a pre-activation vector $a \in \mathbb{R}^H$ and a context vector $s \in \mathbb{R}^S$, the post-activation output $y \in \mathbb{R}^H$ is computed as: $$y = g \odot \rho(a)$$ where $\rho$ is a base nonlinearity (e.g., $\tanh$), $\odot$ is the element-wise product, and $g \in (0,1)^H$ is a context-dependent gate computed via a learnable mapping $G$: $$g = \sigma(G([a; s]))$$ Here, $\sigma$ is the logistic sigmoid function and $[\cdot;\cdot]$ denotes vector concatenation. For theoretical analysis, $G$ is primarily considered an affine transformation: $G([a; s]) = W_g^{(a)} a + W_g^{(s)} s + b_g$. Key Theoretical Contributions This paper provides a comprehensive theoretical analysis of the CAF framework, including: Mathematical Formalization: We establish rigorous definitions and prove fundamental properties such as smoothness and differentiability, which ensure that CAFs are well-suited for gradient-based optimization. Gradient Flow Analysis: We derive the explicit Jacobian structure, establish bounds on its spectral norm to guarantee stable gradient propagation, and analyze conditions for vanishing or amplifying gradients. The Jacobian is decomposed as: $$J_{y,a} = \text{diag}(g) \text{diag}(\rho'(a)) + \text{diag}(\rho(a)) \text{diag}(\sigma'(z)) W_g^{(a)}$$ Expressivity and Universal Approximation: We prove that CAF layers strictly generalize standard activation layers and demonstrate that they retain universal approximation capabilities, offering enhanced representational power for context-dependent functions. Architectural Comparison: We formally distinguish CAFs from related conditioning mechanisms such as Feature-wise Linear Modulation (FiLM), HyperNetworks, and the gates in recurrent architectures (LSTMs/GRUs), highlighting the unique properties of activation-level modulation. Complexity Analysis: We establish that the computational and parametric overhead of CAFs scales quadratically with the hidden dimension but only linearly with the context dimension, ensuring the method remains tractable The framework provides a rigorous foundation for building adaptive neural architectures for temporal modeling, neuromorphic computing, and other domains requiring context-sensitive information processing