A Dual-Transformer Based Model for Biomedical Named Entity Recognition Using PubMedBERT, RoBERTa and CRF

Biomedical Named Entity Recognition (BioNER) plays a key role in processing unstructured medical documents. In this paper, we explore a basic deep learning approach using transformer-based models to identify entities like diseases and genes in biomedical text. The models are evaluated using two standard datasets. The results show satisfactory performance and suggest that transformer models can be useful for basic biomedical text mining tasks. Some simple techniques like character-level features and sequence labeling are also used to improve prediction. This work can be extended in future by using more data and additional features. Biomedical Named Entity Recognition (BioNER) plays a vital role in biomedical text mining by extracting meaningful entities such as diseases, genes, and chemicals from unstructured textual sources. Despite significant advancements, challenges like noisy data, domain-specific terminology, and limited generalization remain. This paper presents a simplified dual-transformer model combining PubMedBERT and RoBERTa to enhance entity recognition in biomedical literature. PubMedBERT captures domain-specific features, while RoBERTa contributes general linguistic context. The outputs of both models are fused using attention-based concatenation and decoded using a Conditional Random Field (CRF) layer to ensure consistent entity labeling. Noise-aware data augmentation techniques are incorporated to improve robustness against misspellings and variations. The model is evaluated on benchmark datasets—NCBI Disease and BC2GM—and achieves a macro F1-score of 90.06% on the test set and 90.98% on the validation set, demonstrating reliable recognition of multi-token biomedical entities and domain-specific abbreviations. The results validate the effectiveness of combining domain-specific and general-purpose transformers in a lightweight framework suitable for real-world biomedical applications.