Toward better semantic segmentation by retaining spectral information using matched wavelet pooling

Abstract Pooling operations, such as average pooling, strided convolution, and max pooling, have become fundamental components of convolutional neural networks (CNNs) due to their ability to capture local features, expand receptive fields, and reduce computational costs. However, in the context of semantic segmentation, these pooling techniques can lead to the loss of crucial spatial details that are necessary for accurate pixel-level predictions. To tackle this issue, extensive research has focused on refining deep CNN models through architectural adaptations and novel training methods. Recent studies have demonstrated the importance of pooling layers, exemplified by innovations like the introduction of wavelet pooling. In our study, we highlight the value of incorporating our previously proposed matched wavelet pooling (MWP) into CNNs to enhance semantic segmentation pipelines. The core concept of MWP challenges the notion that including all sub-bands generated from wavelet decomposition consistently improves accuracy. Instead, we advocate for selecting specific sub-bands for the pooling process in each image during both training and testing. This approach introduces sub-band selection protocols customized for image-specific pooling, designed specifically for semantic segmentation CNN architectures, with a particular focus on the UNet and SegNet models. Across three widely used datasets, our proposed MWP- based pipeline, featuring the MWP-UNet architecture, consistently outperforms conventional pooling methods. It achieves a significant average improvement in intersection over union (IoU) of over 25% compared to recent literature. Additionally, our MWP-SegNet model outperformed the standard SegNet by 12.5% mIoU, further demonstrating the effectiveness of our matched wavelet pooling approach across different network architectures.