An empirical benchmark of activation functions for medical image restoration networks

Abstract Activation functions (AFs) play a pivotal role in defining the representational power and learning dynamics of deep neural networks (DNNs). However, their influence on medical image restoration (MIR) remains insufficiently explored. This empirical study presents a comprehensive, multi-phase investigation into how different AFs affect spatial and frequency characteristics across MIR tasks, such as image deblurring, image deblocking, and image super-resolution, in three medical imaging modalities: computed tomography (CT), digital mammography (DM), and magnetic resonance imaging (MRI). The study integrates a theoretical taxonomy of AFs, a fixed-weight analysis, and a full-scale empirical evaluation involving 15 representative AFs spanning classical, rectified, exponential, and modern smooth families, benchmarked across 3,000 medical images. These functions are implemented within two widely used state-of-the-art MIR architectures: U-Net and Enhanced Deep Super-Resolution (EDSR). Model performance is evaluated using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Edge Preservation Index (EPI), High-Frequency Energy Ratio (HFER), and Learned Perceptual Image Patch Similarity (LPIPS), with statistical validation performed using Analysis of Variance (ANOVA) and Tukey’s Honestly Significant Difference (HSD) tests. Results reveal that while the Rectified Linear Unit (ReLU) and its variants remain strong baselines due to their stability and efficiency, smoother and adaptive functions such as the Gaussian Error Linear Unit (GELU), Mish, and Funnel ReLU (FReLU) achieved superior performance in preserving fine spatial and frequency details. FReLU improved PSNR by 7.6-−9.3 dB over the ReLU baseline across all three modalities in the EDSR architecture (Tukey HSD, $$p<0.001$$ ; Cohen’s $$d = 3.2$$ –4.4), with the largest gains concentrated in the high-spatial-frequency band relevant to fine structures such as mammographic microcalcifications. These findings establish a clear and interpretable link between the mathematical properties of AFs and their mechanistic impact on MIR. The study provides evidence-based guidance for selecting AFs, bridging theoretical understanding and empirical performance to inform the design of efficient, high-fidelity deep learning-based MIR networks.