An OSCE-Inspired Framework for Fine-Tuning Multimodal LLMs in Medical Diagnosis

We present COGNET-MD-X-REQ: a COGnitive NETwork Evaluation Toolkit that identifies precise fine-tuning needs for Large Language Models (LLMs) in multimodal medical diagnosis. COGNET-MD-X-REQ is a novel, two-step evaluation framework, inspired by Objective Structured Clinical Examinations (OSCEs), that enhances LLM applicability and precision. Our framework integrates IoT-driven data retrieval with structured interaction evaluation and domain-specific analysis. Leveraging Image-Metadata Analysis, Named Entity Recognition, and Knowledge Graphs, COGNET-MD-X-REQ identifies weak performance areas by analyzing multimodal model outputs across medical subdomains. We selected GPT-4V to use in the evaluation of our proposed framework on a set of publicly available image-based MCQs in General Pathology. The model achieved 84% accuracy, with notable weaknesses in cardiovascular conditions like atherosclerosis. The framework pinpointed domain-specific deficiencies, enabling targeted fine-tuning. The evaluation leads to the conclusion that our proposed COGNET-MD-X-REQ framework introduces a precision-focused, iterative approach to fine-tuning LLMs by dynamically identifying under-performing areas or knowledge domains that can be related to organs or/and specific conditions and health states. This method reduces reliance on broad retraining, making it suitable for resource-sensitive and safety-critical domains such as medical diagnostics.