Parameter-Efficient Nepali Named Entity Recognition with LoRA and Hindi Cross-Lingual Transfer

Named Entity Recognition (NER) for low-resource languages such as Nepali remains challenging due to limited annotated data and the high computational cost of fine-tuning large pretrained models. In this work, we propose a parameter-efficient approach using Low-Rank Adaptation (LoRA)(Hu et al., 2021) combined with cross-lingual transfer from Hindi. Our optimized method achieves 81.87% ± 0.81% F1 on the EBIQUITY benchmark while training only 2,659,591 parameters (1.50% of the full model), representing a 98.50% reduction compared to full fine-tuning. We conduct a systematic ablation study across multiple configurations to analyze the effects of cross-lingual transfer, training strategy, adapter capacity, target module selection, and PEFT method choice. Our results show that (1) Hindi transfer improves performance by +2.85% F1 over Nepali-only training, (2) joint training outperforms sequential fine-tuning by 4.2% F1 due to catastrophic forgetting, (3) LoRA applied to all attention modules outperforms query-value only by +2.62% F1, and (4) LoRA outperforms all other PEFT baselines including BitFit and Prefix Tuning by large margins. Despite a modest performance gap compared to full fine-tuning (Yadav et al., 2024), our approach achieves over 60× faster training with significantly reduced computational requirements, making it suitable for resource-constrained NLP research. We release our code and trained models to support future work in Nepali and other low-resource languages at https://github.com/a-proton/nepali-ner-lora and https://huggingface.co/a-proton/nepali-ner-lora.