Comparing traditional NLP methods and LLM-based extraction for identifying biomarkers in lung cancer.

e13607 Background: Lung cancer treatment relies heavily on genomic biomarkers, which are often recorded in unstructured documents like clinical notes. Extracting and interpreting this data can help identify eligible patients for clinical trials. Traditional Named Entity Recognition (NER) models, such as Regex and Long Short-Term Memory (LSTM), have been useful in identifying entities, Small Language Models (SLMs) and Large Language Models (LLMs) based NER have shown promise in handling complex and variable information. The objective of this study was to compare these two methods for extracting genomic data from oncology notes in the EHRs. Methods: We analyzed 27 de-identified clinical notes from 29 lung cancer cases with biomarker data. The notes were sourced from different hospitals, after ethical committee approval, ensuring variability in documentation. Notes were processed by "SLM & LLM-based NER model" and "pre-transformed traditional NER." F1 scores of both the models were compared for clinically relevant attributes of genomic markers, such as categorical results, exonic location, variant type, and genomic alterations. Results: The SLM and LLM based NER model outperformed the traditional NER model in identifying the biomarker entity, variant type and categorical results (Table). In addition, a qualitative assessment of other attributes like exon location and genomic alterations which were not available through traditional NER models and could be extracted satisfactorily through the SLM and LLM based NER model for e.g. MET exon 14 and EGFR genomic alteration had F1 score of 0.8 and 0.75, respectively. Conclusions: In precision oncology, identifying biomarker variants is crucial for targeted interventions. Clinical notes are a rich source of patient information including genomic data, making them key evidence to enrich the database. Our study demonstrates that SLM and LLM based NER models are better at distinguishing contextual information, improving their ability to perform precise information extraction, such as differentiating between 'EGFR' as a biomarker and 'eGFR’ as lab test and hence can significantly aid in extracting precision oncology data from unstructured clinical notes. This approach enhances the ability to support personalized treatments and clinical trials. Variable Biomarker name Traditional NER model Biomarker nameSLM & LLM based NER Variant type Traditional NER model Variant type SLM & LLM based NER Categorical result Traditional NER model Categorical result SLM & LLM based NER EGFR 0.85 0.98 0.04 0.87 0.38 0.78 ALK 0.89 0.99 0.03 0.97 0.50 0.86 ROS1 0.86 0.98 0.02 0.83 0.55 0.84 KRAS 0.85 0.97 0.02 0.90 0.46 0.52 BRAF 0.89 0.99 0.04 0.92 0.51 0.90 HER2 0.84 0.98 0.14 0.86 0.65 0.97 MET 0.81 0.96 0.02 0.83 0.60 0.83 RET 0.82 0.96 0.01 0.79 0.62 0.77