186 CNS-CLIP: Transforming a Neurosurgical Journal Into a Multimodal Medical Model

INTRODUCTION: Classical biomedical data science models are trained on a single modality and aimed at one specific task. However, the exponential increase in the size and capabilities of the foundation models both inside and outside of medicine shows a shift towards task-agnostic models utilizing large-scale, often internet-based, data. Recent research into smaller foundation models trained on specific literature, such as programming textbooks, demonstrated that they can display capabilities similar to, or superior to, large generalist models, suggesting a potential middle ground between small task-specific models and large foundation models. METHODS: We constructed a multimodal dataset from Neurosurgery Publications articles, involving PDF data collection and figure-caption extraction with an AI quality control pipeline. We then developed a fine-tuning protocol for the OpenAI CLIP model using this dataset. We evaluated the model using neurosurgical information retrieval, a CT imaging classification, and generalist zero-shot ImageNet classification tasks. RESULTS: Our final dataset included 24,021 figure-caption pairs. CNS-CLIP demonstrated superior performance in neurosurgical information retrieval with a Top-5 accuracy of 48.84%, compared to 18.03% for the baseline. The average area under ROC across six neuroradiology tasks achieved by CNS-CLIP was 0.95, slightly superior to OpenAI’s CLIP at 0.94, and both significantly outperforming Vision Transformer at 0.62. In generalist classification, CNS-CLIP reached a Top-5 accuracy of 75.99%, a decrease from the baseline of 80.20%, demonstrating catastrophic forgetting phenomenon. CONCLUSIONS: This study presents a pioneering effort in building a domain-specific multimodal model using data from a medical society publication. The results indicate that domain-specific models, while less globally versatile, can offer advantages in specialized contexts. This emphasizes the importance of using tailored data and domain-focused development in training foundation models in neurosurgery specifically and medicine in general.