Open-set semantic segmentation for remote sensing images

Collecting samples that exhaust all possible classes for real-world tasks is usually difficult or impossible due to many different factors. In a realistic/feasible scenario, methods should be aware that the training data is incomplete and that not all knowledge is available. Therefore all developed methods should be able to identify the unknown samples while correctly executing the proposed task to the known classes in the tests phase. Open-Set Recognition and Semantic Segmentation models emerge to handle this kind of scenario for, respectively, visual recognition and dense labeling tasks. Initially, this work proposes a novel taxonomy aiming to organize the literature and provide an understanding of the theoretical trends that guided the existing approaches that may influence future methods. This work also proposes two distinct techniques to perform open-set semantic segmentation. First, a method called Open Gaussian Mixture of Models (OpenGMM) extends the Open Principal Component Scoring (OpenPCS) framework using a Gaussian Mixture of Models to model the distribution of pixels for each class in a multimodal manner. Second, the Conditional Reconstruction for Open-set Semantic Segmentation (CoReSeg) method tackles the issue using class-conditioned reconstruction of the input images according to their pixel-wise mask. The third proposed approach is a general post-processing procedure that uses superpixels to enforce highly homogeneous regions to behave equally, rectifying erroneously classified pixels within these regions. We also proposed a novel superpixel generation method called Fusing Superpixels for Semantic Consistency (FuSC). All proposed approaches produce better semantic consistency and outperformed state-of-the-art baseline methods on Vaihingen and Potsdam ISPRS dataset. The official implementation of all proposed approaches is available at https://github.com/iannunes.