Fast fourier transform gated activation function (FFTGate)

• FFTGate: a novel activation function that integrates temporal and spectral activation histories. • FFT-based frequency-domain gating modulates neuron activations using spectral features. • Target-adaptive regularization of the activation’s trainable parameters improves generalization. • FFTGate improves generalization on image classification and neural machine translation tasks. • FFTGate outperforms 18 existing activation functions on benchmark datasets. Activation functions are crucial in determining how deep neural networks process information and adapt during training. Although widely adopted activation functions such as ReLU, GELU, and Swish demonstrate strong empirical performance, they rely primarily on pre-activation inputs, ignoring temporal activation histories that capture how neuron activations evolve over time and spectral features that capture frequency-domain patterns in activation histories. This can limit their adaptability, particularly in dynamic or time-varying tasks. To address this, we propose FFTGate , a novel activation function that integrates temporal and spectral activation information from neuron activation histories into the activation process. Specifically, FFTGate first captures per-channel activation histories using a sliding window of a defined length. Next, it applies the Fast Fourier Transform to extract spectral features from the captured activation histories, which are then used to form an FFT-based gating mechanism subsequently used to modulate each neuron’s pre-activation input. This enables networks to adapt more effectively to evolving inputs, thereby improving generalization. FFTGate is evaluated on both vision and language domains. Experimental results show that FFTGate consistently outperforms 18 established and recently proposed activation functions on small-scale image classification tasks (CIFAR-10, CIFAR-100; Top-1 accuracy gain of +0.74 %) and large-scale image classification task (ImageNet-1K; Top-1 accuracy gain of +0.36 %), demonstrating strong generalization on datasets of different scales. Furthermore, FFTGate achieves even greater improvements in neural machine translation, with BLEU score gains of +2.94 on Kaggle German–English, +1.08 on IWSLT’13 French–English, and +0.88 on IWSLT’13 English–French over the best-performing baselines. These results demonstrate that FFTGate enhances generalization and stability in both image-based and time-varying tasks. Code and results are available at: https://github.com/Ndupuechi/FFTGate .