Enhancing Out of Vocabulary(OOV) Token representation with Aggregated Conditional Random Fields model for Kannada NER

Named entities are dynamic, as the contexts shift in a language, new entities emerge, reflecting the change over time, but large language models have a fixed vocabulary making it difficult to process out-of-vocabulary words(OOV), there are deep learning methods like BiLSTM, attention mechanism which can be used to get great results for Named entity recognition, but in case of low resource languages like Kannada, data available is not sufficient to train these deep learning models, so the research work attempts to solve the problem of NER using Aggregated Conditional Random Fields. The training is done on ai4bharat/naamapadam dataset which is the largest publicly available dataset for 11 major Indian languages including Kannada. The tokenization is done using a transformer-based multilingual model xlm-roberta-base. In this experiment, the OOV words are split into multiple sub-tokens and then aggregated using an additive pooling layer, and then passed on to the CRF classifier. Results show that the additive pooling model performs better than the base model benchmark and also better than vanilla CRF. Our model was able to produce an f1 score of 78.97 containing around 5728 OOV words in test and 14545 OOV words in the validation set.