A Hybrid Framework Combining Dictionary-Based Methods, BERT, and CRF for Language Identification and Normalization in Code-Mixed Hinglish

A hybrid model for language identification and normalization offers the advantage of combining the strengths of both machine learning and rule-based approach which provide flexibility and scalability in handling diverse and complex language patterns, especially in code-mixed text. This paper proposes a novel hybrid approach for Language Identification (LI) and normalization in code-mixed Hinglish text, combining dictionary-based methods with Bidirectional Encoder Representations from Transformers (BERT) embeddings and Conditional Random Fields (CRF). The approach starts with a dictionary lookup for initial tagging of standard vocabulary, providing a solid foundation for accurate language detection. CRF model is used, which considers word dependencies and performs very well on the sequence labelling task. The outputs are combined with a weighted ensemble which is tuned for the optimaltrade-off between recall and precision. This approach ensures both normalization of Hinglish and enhanced LI accuracy while preserving linguistic and culturalnuances. As a first, it takes the challenges of code-mixed languages into consideration, demonstrating a significant progression in processing multilingual text, which captures the constantly evolving nature of code-mixed languages.