G2T: Gate guided transformer for medical image segmentation

Abstract Inspired by the human cognitive process and memory capabilities, we present a novel vision transformer (ViT) in this paper, called G ate G uided T ransformer (G2T) for medical image segmentation. G2T addresses a key limitation of cascaded decoder blocks in vision transformers; that is their difficulty in retaining and propagating important information from earlier stages, which often leads to suboptimal feature decoding in later blocks. Our proposed G2T has (i) a novel feature-aware decoder (FAD) consisting of three convolutional gates named Refine, Expand, and Produce (REP), which deal with forgetting irrelevant features, remembering relevant features, and generating the output features, respectively; (ii) a novel attention mixing module called LH-RAAM (Low to High-Resolution Attention Accumulation Module) that accumulates attentions computed in the encoder blocks from low-level features to high-level features, and aggregates them with the attentions computed in the decoder blocks to emphasize similar features during both the encoding and decoding stages. Evaluated on six medical imaging datasets, G2T consistently outperforms existing state-of-the-art models by 1-3%, achieving a 94.67% mean Dice score ±0.01 standard deviation on the ACDC dataset and 92.90% ±0.02 on the Synapse dataset. Ablation studies additionally justify the utility of each component of G2T along with strong generalizability outside the medical domain.