Evaluating Transformer Models for Named Entity Recognition in Indian Legal Texts

Named Entity Recognition (NER) in legal texts is vital for extracting information from vast, unstructured legal documents. The complexity of legal language—characterized by formal style, domain-specific terminology, and unique entity types such as case numbers, legal citations, court names, judges, and involved parties—poses significant challenges. Traditional machine learning models fall short due to their reliance on manual feature engineering and limited adaptability to the nuanced legal language. This paper evaluates transformer-based models, DistilBERT, DistilRoBERTa, and InLegalBERT, for NER in Indian court judgments using a comprehensive dataset prepared by OpenNyAI, which contains 46,545 annotated legal entities across 14 distinct types (e.g., petitioner, respondent, statute). To address imbalanced entity distribution, we propose a cleaning process and optimization of dataset splits using Jensen-Shannon Divergence, ensuring a more balanced representation of entities across training, validation, and test sets. We fine-tuned each model on three versions of the dataset: (1) the original dataset, (2) a cleaned dataset with sentences lacking tagged entities removed, and (3) a version with optimized entity distribution. Performance evaluation based on precision, recall, and F1-score revealed that DistilRoBERTa achieved the highest F1-score of 0.8672, significantly outperforming DistilBERT and demonstrating improvements over InLegalBERT, which is pre-trained on legal texts. Our results underscore that balanced dataset splits and robust preprocessing are crucial for enhancing model generalizability and performance in legal NER tasks, as evidenced by substantial gains in F1-scores following these preprocessing steps.