Metric-Based Techniques for Reducing Out-of-Vocabulary Rates in Romanized Text Processing

In this research work, custom metrics have been constructed for measuring out-of-vocabulary (OOV) words of Romanized language models that include Bhojpuri, Hinglish and Punjabi Romanized text input. The end purpose of these metrics is to provide a framework for handling and improving the quality of language models, the work has been demonstrated on three Romanized language models that include Punjabi, Hindi and Bhojpuri. For all this, new datasets were curated and approach revolved around covering a comprehensive a suite of metrics, such as OOV Ratio, IV Ratio, OOV Rate, Vocabulary Coverage and OOV Density, providing a robust mechanism to identify and mitigate OOV issues across multiple categories of words and sentences. Using the negation of negation approach, demonstrate significant reductions in OOV word ratios and enhancements in the performance of models in question. For the Bhojpuri model, the OOV Ratio was reduced from 41.18% to 10%, IV Ratio increased from 58.82% to 90% and Vocabulary Coverage improved from 58.82% to 90%. At the same instance, the Hinglish model’s OOV Ratio decreased from 32.18% to 8%, with an IV Ratio improvement from 50.12% to 92% and Vocabulary Coverage reaching 92%. The Punjabi Romanized model saw its OOV Ratio drop from 52.18% to 10%, IV Ratio rise from 50.12% to 90% and Vocabulary Coverage improves to 90%.