Semantic segmentation of terrestrial whole-sky images using the new W-Net model with the stationary wavelet transform 2D

This work proposes a novel deep learning model, named W-net, focused on the semantic segmentation of whole-sky images obtained by fisheye cameras. The model is based on the use of two U-net networks connected in series, interlinked by skip connections and attention skip connections. Additionally, the proposed approach incorporates a color space transformation layer that converts images from the RGB space to either HSV or CIE XYZ, followed by a feature extraction layer utilizing the 2D Wavelet Transform. Novel attention mechanisms are introduced, notably the one responsible for the transition of information between the two U-nets. To evaluate the model’s performance, a comparative analysis was conducted against four well-established models in the literature. It is noteworthy that, while three of these models are designed for binary semantic segmentation, considering only the “sky” and “cloud” classes, the W-net model employs multiclass semantic segmentation, differentiating among the “sky”, “sun”, “cloud” and “edge” categories. Experimental results demonstrate the superiority of the W-net architecture. The unweighted version achieved a Mean Intersection over Union (MeanIoU) of 87.63%, a Dice coefficient of 96.30%, an overall Accuracy of 97.40%, and a Precision of 93.07%. The weighted W-net further improved the results, achieving a MeanIoU of 87.79%, a Dice coefficient of 96.62%, an Accuracy of 97.41%, and a Precision of 89.89%. These outcomes confirm that the proposed model outperforms the benchmark methods, and that the inclusion of weighting enhances the detection of sun regions. Finally, a qualitative evaluation was performed through a visual comparison between the manually annotated masks and those generated by the proposed model. • A novel deep learning model is introduced, combining two U-net architectures with attention-based skip connections for semantic segmentation of whole-sky images. • The proposed approach addresses the scarcity of multiclass segmentation models in the context of sky imagery by distinguishing between sky, sun, cloud, and edge regions. • New attention mechanisms and a wavelet-based feature extraction layer are integrated to enhance feature representation and information flow between the U- net stages.