Leveraging Contextual Information in Biomedical Named Entity Recognition: Determining the Optimal Depth of Context

The significance of Biomedical Natural Language Processing (Bio-NLP) has escalated in the current era, owing to the rapid expansion of biomedical data. Biomedical Named Entity Recognition (Bio-NER) is a fundamental task in Bio-NLP that involves the identification and categorization of named entities within biomedical text documents. During the early days, various traditional, machine learning, and deep learning-based methods were proposed to solve the Bio-NER task. However, each of these methods has its own limitations and drawbacks. With the advent of transformer-based pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT), Bio-NER methods advanced significantly through fine-tuning approaches. These approaches leverage the learned weights of pre-trained language models as a starting point for Bio-NER. BioBERT is a specialized adaptation of BERT in the biomedical domain, which learns word representations from both left and right contexts. In addition, PubMedBERT shares the same architecture as BERT; however, unlike BioBERT, it is trained from scratch on PubMed abstracts, ensuring domain-specific pretraining for improved biomedical text understanding. To explore the importance of context information, we propose an effective method for Bio-NER using BioBERT and PubMedBERT models by incorporating more context information during fine-tuning (referred to as CI-BioBERT and CI-PubMedBERT). We perform a comprehensive analysis that encompasses models enriched with diverse forms of context information, including self-sentence and cross-sentence contexts: (i) previous sentence only, (ii) next sentence only, (iii) both previous and next sentences, (iv) next sentences up to the maximum length, and (v) previous sentences up to the maximum length. Further, the effectiveness of the proposed model is experimentally analyzed on the NCBI disease, BC5CDR disease, and BC5CDR chemical datasets under various hyperparameter settings. The experimental results reveal that incorporating contextual information improves the performance of Bio-NER. Since ensemble learning can further enhance performance, we employed ensemble learning to combine predictions from the top three performing models based on contextual information. The experimental results demonstrate that the proposed approach outperforms competitive methods in terms of the F1 score and Recall.