Spectral Wavelet Dropout: Regularization in the Wavelet Domain

Regularization techniques help prevent overfitting and therefore improve the ability of convolutional neural net-works (CNNs) to generalize. One reason for overfitting is the complex co-adaptations among different parts of the network, which make the CNN dependent on their joint response rather than encouraging each part to learn a useful feature representation independently. Frequency domain manipulation is a powerful strategy for modifying data that has temporal and spatial coherence by utilizing frequency decomposition. This work intro-duces Spectral Wavelet Dropout (SWD), a novel regularization method that includes two variants: ID-SWD and 2D-SWD. These variants improve CNN generalization by randomly dropping detailed frequency bands in the discrete wavelet decomposition of feature maps. Our approach distinguishes itself from the pre-existing Spectral “Fourier” Dropout (2D-SFD), which eliminates coefficients in the Fourier domain. Notably, SWD requires only a single hyperparameter, unlike the two required by SFD. We also extend the literature by implementing a one-dimensional version of Spectral “Fourier” Dropout (lD-SFD), setting the stage for a comprehensive comparison. Our evaluation shows that both ID and 2D SWD variants have competitive performance on CIFAR-IO/IOO benchmarks relative to both ID-SFD and 2D-SFD. Specifically, ID-SWD has a significantly lower computational complexity compared to ID/2D-SFD. In the Pascal VOC Object Detection benchmark, SWD variants surpass ID-SFD and 2D-SFD in performance and demonstrate lower computational complexity during training.