Automated Ultrasound Diagnosis via CLIP-GPT Synergy: A Multimodal Framework for Image Classification and Report Generation

As a crucial non-invasive imaging modality in clinical diagnosis, ultrasound interpretation faces challenges of subjectivity and inefficiency. To address the limitations of traditional single-modal deep learning models in cross-modal alignment and structured text generation, this study proposes an intelligent analysis system based on a CLIP-GPT joint framework, integrating contrastive learning with generative pre-training for end-to-end image classification and diagnostic report generation. Utilizing an ultrasound dataset containing six types of liver lesions, we implement a multi-stage training strategy: first establishing visual-semantic cross-modal mapping through CLIP, followed by fine-tuning GPT-3.5 to construct a structured report generator. Experimental results demonstrate superior performance in classification accuracy (96.4%) and report generation quality (ROUGE-L 41.2, BLEU-4 32.5) compared to conventional models. Key innovations include: a cross-modal feature decoupling-recombination mechanism bridging semantic gaps, clinical guideline-driven hierarchical templates ensuring professional standardization, and dynamic attention strategies enhancing lesion discrimination. This study provides an interpretable multimodal solution for medical image analysis, offering significant clinical value for intelligent diagnostic systems.