TAFSIL: Taxonomy Adaptable Fine-grained Entity Recognition through Distant Supervision for Indian Languages

Several studies have used distant supervision to create resources for fine-grained entity recognition (FgER) to mitigate the challenges of manual annotation.However, most of these methods are primarily developed for English and cannot be efficiently adapted to many other languages, including Indian languages.Moreover, the emergence of new and unseen entity types deteriorates the performance of the supervised models trained on FgER datasets with different predefined sets of entity types.This work introduces TAFSIL, a taxonomy-adaptable FgER framework to create FgER datasets in six Indian languages.The chosen languages are spoken by more than a billion speakers across various countries.TAFSIL utilizes the high interlink between the knowledge base WikiData and linked corpora Wikipedia through multi-stage heuristics and improves annotation through fuzzy match and quality sentence selection.TAFSIL enables us to create datasets of a total size of around three million samples for six languages Hindi (Hi), Marathi (Mr), Sanskrit (Sa), Tamil (Ta), Telugu (Te), and Urdu (Ur) belonging to two language families Indo-European and Dravidian.We evaluate the robustness of TAFSIL by creating various datasets in four taxonomies FIGER, OntoNotes, HAnDS, and MultiCoNER2.Our extensive experiments suggest the sound quality of the datasets as there is a relative improvement of 83% in average F1 score over zero-shot performance across various FgER state-of-the-art models.The resource is publicly available at https://huggingface.co/datasets/prachuryyaI-ITG/TAFSIL.