Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function

Abstract Training of semantic segmentation models for material analysis requires micrographs as the inputs and their corresponding masks. In this scenario, it is quite unlikely that perfect masks will be drawn, especially at the edges of objects, and sometimes the amount of data that can be obtained is small, since only a few samples are available. These aspects make it very problematic to train a robust model. Therefore, we demonstrate in this work an easy-to-apply workflow for the improvement of semantic segmentation models of micrographs through the generation of synthetic microstructural images in conjunction with masks. The workflow only requires joining a few micrographs with their respective masks to create the input for a Vector Quantised-Variational AutoEncoder (VQ-VAE) model that includes an embedding space, which is trained such that a generative model (PixelCNN) learns the distribution of each input, transformed into discrete codes, and can be used to sample new codes. The latter will eventually be decoded by VQ-VAE to generate images alongside corresponding masks for semantic segmentation. To evaluate the quality of the generated synthetic data, we have trained U-Net models with different amounts of these synthetic data in conjunction with real data. These models were then evaluated using real microscopic images only. Additionally, we introduce a customized metric derived from the mean Intersection over Union (mIoU) that excludes the classes that are not part of the ground-truth mask when calculating the mIoU of all the classes. The proposed metric prevents a few falsely predicted pixels from greatly reducing the value of the mIoU. With the implemented workflow, we were able to achieve a time reduction in sample preparation and acquisition, as well as in image processing and labeling tasks. The idea behind this work is that the approach could be generalized to various types of image data such that it serves as a user-friendly solution for training models with a smaller number of real images.