Multi-Aperture Fusion of Transformer-Convolutional Networks With Curvature-Aware Loss Function Enhances 3D Segmentation of Clinical and Microscopy Images

Automated 3D Radiological and Microscopy image analysis enables quantitative therapeutics assessment. However, accurate segmentation and smaller footprints are needed. To address these issues, the current study introduces the Multi-Aperture Transformer-Convolutional Network (MAFTC-Net) with a custom loss function incorporating principal curvatures to penalize topological dissimilarities. The multi-aperture modality represents each image patch as a series of images with smaller apertures along the same central field of view. Each aperture has a pyramid representation, with the advantage of maintaining the original image resolution. The MAFTC-Net encoder comprises four Swin Transformers, where a convolutional network processes the output of each transformer to accentuate local features within a global context. Next, the outputs of the transformer and convolution blocks are integrated with a 3D fusion block. MAFTC-Net was trained on the Synapse multi-organ, ACDC, and 3D Organoid datasets, achieving an average Dice score of 90.18±0.04, 93.55±0.05, and 95.13±0.03, respectively. The improved performance also comes with the added benefits of reduced complexity with 38.2 million parameters.