Integrating Large Language Models with Deep Learning for Medical Imaging Modality Classification

This study proposes an automated system for medical image modality classification that combines large language models (LLMs) and a custom deep learning architecture.First, the Llama 3.2 model was used to label the textual descriptions from the radiology objects in context (ROCO) dataset, which contained 81,000 medical images.The extracted labels and their corresponding images were then used to train a specialized image classification model consisting of residual and squeeze-andexcitation (SE) blocks with attention mechanisms.To ensure fair comparison and consistency, all samples were preprocessed, the data were augmented, and the class balance was maintained.The experiments revealed that the SEBwithAttention model obtained better results, with 88.100.145%accuracy and an F1-score of 83.550.295,which was higher than those of DenseNet, ResNet50, VGG16, AlexNet and vision transformers.In conclusion, the study findings demonstrated that LLM-based modality extraction could effectively classify medical images using their captions, enabling the training of a model that accurately identified image modalities.