An Explainable Multimodal Deep Learning Framework for Thyroid Nodule Diagnosis in Ultrasound Imaging Using Hybrid Vision Transformers and Med-PaLM

Thyroid tumors rank among the most frequently occurring endocrine cancers because early detection helps doctors deliver effective treatments that lead to better patient results. Ultrasound imaging enables the detection of thyroid nodules, yet medical professionals struggle to differentiate between benign and malignant nodules through their diagnostic tests. This study introduces a new medical framework that enables thyroid nodule diagnosis through ultrasound imaging. The proposed model combines advanced segmentation with feature extraction, classification, and reasoning components to create a complete system. The specialized segmentation method shows accurate results when it detects nodule boundaries, which leads to better analysis of specific regions. The Hybrid Vision Transformer (HVT) operates to capture detailed textural information together with complete environmental patterns, which boosts its ability to classify different elements. The proposed framework incorporates a Large Language Model (LLM), specifically Med-PaLM, to provide context-aware clinical reasoning and interpretation. The structured evaluation process uses Thyroid Imaging Reporting and Data System (TI-RADS)-based feature scoring to compare model results with designated clinical standards. The diagnostic process is enhanced through the use of a language model, which delivers contextual understanding and produces valuable information from features that have been extracted. The proposed model achieves excellent performance with accuracy at 98.5%, precision at 98.7%, recall at 98.4%, and F1-score at 98.5%, which demonstrates its capacity for accurate and equivalent performance across different classifications. The experimental results demonstrate that the model achieves better results than existing methods. The combination of multimodal data with clinical reasoning improves both the accuracy and the user experience of the system. The proposed framework provides an efficient, interpretable, and scalable solution for thyroid nodule diagnosis.