TTT-Vnet: a 3D vision test-time training model for medical image analysis

We propose a novel V-shape three-dimensional vision test-time training (TTT) Vnet, to enhance deep-learning-based medical image analysis. Recent advancements have seen convolution neural networks (CNNs) and vision Transformers (ViT) excel in segmentation, synthesis, and registration tasks. However, CNNs and ViT cannot effectively and simultaneously capture local and global features. Moreover, they tend to overfit data due to limited medical image datasets. Inspired by TTT’s generalization capabilities, our approach synergizes CNN and TTT to capture both local and global features. The TTT-Vnet, featuring an encoder-decoder architecture with shifted-window (Swin) mechanism and TTT modules in the deep layers, captures highly compressed, generalizable global features, enhancing learning for various medical tasks. Unlike conventional CNN/ViT models, TTT layers continue to learn and adapt to each new image sample during testing time, improving understanding of specific medical image samples and demonstrating better generalization. We implement the proposed TTT-Vnet model into different deep-learning frameworks to demonstrate the potential of using the proposed TTT-Vnet to achieve computed tomography (CT)-based organ segmentation, T1-weighted magnetic resonance imaging (MRI) synthesis, and 4DCT image registration. We also perform ablation study using CNN-and ViT-based models to benchmark the proposed TTT-Vnet. The results indicate that TTT-Vnet shows superior performance compared to conventional deep-learning networks.