Over neurale architecturen, verliesfuncties en evaluatiemetrieken voor semantische segmentatie

Semantic segmentation is a foundational task in computer vision where each pixel in an image is assigned to a specific class label, essentially grouping pixels based on their semantic meaning. Unlike image classification, which assigns a single label to the entire image, or object detection, which focuses on identifying bounding boxes, semantic segmentation provides a detailed understanding of the scene by labeling each pixel with the category it belongs to. This task is critical in applications such as autonomous driving, robotics, aerial imaging and medical imaging. In the thesis, we delve into the three aspects of semantic segmentation: neural architectures, loss functions and evaluation metrics. Neural architectures in semantic segmentation often build on convolutional neural networks (CNNs) with modifications to capture pixel-level details. One of the most popular architectures is UNet, which incorporate an encoder-decoder structure to balance spatial resolution and semantic depth. Due to its simplicity and effectiveness, UNet is widely used across various applications such as aerial imaging and medical imaging. However, its architecture is often applied without sufficient optimization, relying either on manual tuning or no tuning at all, which can be inefficient and suboptimal. In Chapter 3, we build upon and extend AOWS. Our method enhances AOWS by incorporating (i) a general MRF framework, (ii) diverse M-best loopy inference, and (iii) differentiable parameter learning. This allows for efficient exploration of network architectures that leverage loopy graphs, including skip connections. Using UNet as a baseline, we identify new architectures, MRF-UNets. Experiments demonstrate that our MRF-UNets outperform numerous benchmarks on various aerial imaging and medical imaging datasets, while maintaining low computational complexity. Loss functions in semantic segmentation are typically aligned with evaluation metrics to optimize performance. These tailored losses have been shown to outperform traditional pixel-wise losses, such as the cross-entropy loss, in segmentation tasks. However, we find that current loss functions lack flexibility when incorporating important training techniques like label smoothing, knowledge distillation, and semi-supervised learning, primarily due to their inability to handle soft labels. In Chapter 4 and 5, we propose variants of these loss functions that, while maintaining their equivalence in traditional settings with hard labels, also support soft labels. We apply these loss functions to key scenarios, including label smoothing, knowledge distillation, and semi-supervised learning, demonstrating their ability to boost both model accuracy and calibration. Across multiple semantic segmentation datasets and architectures, our method consistently outperforms the cross-entropy loss, and our straightforward approach surpasses state-of-the-art techniques in knowledge distillation and semi-supervised learning. Evaluation metrics in semantic segmentation primarily focus on measuring the overlap between predicted and ground truth labels. However, semantic segmentation datasets often suffer two types of imbalance: class imbalance, where some classes appear more frequently than others and size imbalance, where some objects occupy more pixels than others. Standard evaluation metrics, such as overall pixel-wise accuracy and mean intersection over union (mIoU), tend to be biased towards majority classes and larger objects. To counteract this, in Chapter 6, we propose fine-grained mIoUs along with the corresponding worst-case metrics, providing a more comprehensive evaluation. These metrics reduce bias towards large objects, offer more nuanced statistical insights, and enable better auditing of models and datasets. Additionally, we conduct an extensive benchmark study, training multiple modern neural networks on diverse segmentation datasets from scratch. This study underscores the importance of using multiple metrics for evaluation and confirms that fine-grained mIoUs can mitigate bias towards large objects. Furthermore, we identify the critical roles of architecture designs and loss functions in optimizing these metrics, leading to best practices for improving fine-grained metric performance. In summary, we aim to address the three aspects of semantic segmentation: neural architectures, loss functions and evaluation metrics. In doing so, we provide new insights and methodologies that push the boundaries of current semantic segmentation techniques, paving the way for more effective and reliable models across various applications.