ViT-radiomics fusion for lymph node classification in ultrasound images: a multicenter study

Abstract Background Ultrasound (US) is widely used for assessing lymph node (LN) status, but its diagnostic accuracy remains highly operator dependent. A robust computer-assisted diagnostic model may enhance clinical performance and improve inter-operator and inter-center consistencies. Purpose To develop and validate a multimodal fusion model, ViT-Rad, that combines radiomics features and deep learning features derived from vision transformers (ViT) for the classification of benign and malignant LNs in US images. Methods Between February 2016 and November 2023, a total of 1647 ultrasound images were retrospectively collected for analysis. In this multicenter study, we constructed ViT-Rad, a three-module neural network integrating ViT-based global contextual features and radiomics features extracted from manually delineated regions of interest. To address potential cross-center domain shift, we further employed weak/strong augmentation and a few-shot domain adaptation strategy using limited labeled external-center samples. Results The model was trained and evaluated on a dataset from Center 1 ( n = 1273; mean ± SD age, 57 ± 14 years), and its generalizability was tested on an external dataset from Center 2 ( n = 374; mean ± SD age, 52 ± 18 years). ViT-Rad achieved an AUC of 0.95 [95% CI 0.91, 0.98] and an accuracy of 0.90 [95% CI 0.85, 0.95] on the internal test set, outperforming conventional radiomics models (AUC = 0.79, 95% CI 0.71, 0.89; P = .006). With domain adaptation, its AUC on the external set increased from 0.73 [95% CI 0.69, 0.79] to 0.85 [95% CI 0.81, 0.90]. These findings suggest improved adaptation-assisted external performance under cross-center domain shift. Conclusion By combining radiomics and ViT-derived features, ViT-Rad effectively integrates domain-specific and global contextual information, improving internal diagnostic performance and showing improved external adaptability after few-shot domain adaptation for LN classification on US images.