Development of a machine learning model for automatic data extraction from breast cancer pathology reports

Data extraction from medical records is crucial for clinical research, with current methods relying on human annotation. Natural Language Processing (NLP) and Machine Learning-based approaches show promise. We develop and evaluate an NLP pipeline constructed by selecting among four candidate models; ClinicalBERT, PubMedBERT, BioMedRoBERTa and Mistral-Nemo LLM to automate data extraction of 1,795 breast cancer pathology reports obtained from the Providence Health Services Authority in British Columbia. We also explore the effect of further pre-training the BERT-based models using the SQuAD question-answering dataset. Accuracy was evaluated by comparing model output and human annotation. PubMedBERT pre-trained on SQuAD proved to be the best performing model, achieving an overall accuracy of 97.4%. 30 of the 32 FOIs had an accuracy greater than 95.0%. Our model outperformed a previous rule-based algorithm (95.6%). Our findings demonstrate how a high-performing question-answering NLP pipeline for breast cancer pathology can provide a scalable approach to high-fidelity extraction of clinicopathologic features, thereby enhancing research efficiency and improving clinical outcomes.