189 A Generalizable Pipeline for Building an Extensive Domain-Specific Dataset From a Medical Journal - Neurosurgery Edition

INTRODUCTION: The advancement of artificial intelligence in medicine has greatly increased the demands for robust, high-quality domain-specific datasets that are critical for developing expert language and vision models, particularly in Neurosurgery. Our project aims to develop a scalable, efficient pipeline to extract full-text PDFs and curate a comprehensive text and image dataset from all literature available from Neurosurgery Publications. METHODS: We automated the online acquisition of full-text PDF files across various CNS journals using Python’s Selenium library. Our designed journal harvester bypasses any dynamic website elements and collects all downloadable article PDFs across all journal’s issues by identifying keywords, icons, and JavaScript elements. This PDF Literature Database was built with metadata and robust error-handling mechanisms that detect interruptions/delays, inaccessible links, duplicate and broken files. Using the database, we used Nougat to extract all texts into markdown format and PyMuPDF to extract images and corresponding captions, which were paired via spatial proximity on the PDF page using the Hungarian Algorithm. A pretrained ResNet model was used to remove irrelevant statistical graphs and charts. RESULTS: From Neurosurgery(1955-2024), Operative Neurosurgery(2005-2024), and Neurosurgery Practice(2020-2024), we obtained all 21219, 2618, and 147 available PDFs, respectively, of open and closed-access articles totalling 23984 PDF files. This is the largest neurosurgery-journal-specific text and image database available, containing 139 million words and 5.3 million sentences, as well as a total of 24,021 image-text pairs after quality control, ready for developing advanced neurosurgical AI models. CONCLUSIONS: We present a scalable pipeline for generating an extensive domain-specific dataset of text and images from academic journals, demonstrated with Neurosurgery Publications. The methodology is easily adaptable across all medical subspecialties, potentially revolutionizing medical AI training by building the most comprehensive database of medical literature through collaborative and legal efforts.