An Efficient Multilingual Text Classification using IndicCorp dataset

Language detection is a crucial preprocessing step in natural language processing (NLP) tasks, especially in a multilingual environment. This paper presents a language detection system utilizing the Naive Bayes classifier, applied to the IndicCorp dataset, which includes a diverse range of languages prevalent in the Indian subcontinent. The IndicCorp dataset provides a rich source of textual data across multiple Indian languages, enabling the development of robust and accurate language models. Our system leverages the simplicity and effectiveness of the Naive Bayes algorithm, known for its high efficiency in text classification tasks. The proposed approach involves pre processing the dataset, extracting features using the CountVectorizer, and training the Naive Bayes classifier to identify the language of a given text. Experimental results demonstrate the system’s capability to achieve an accuracy of 73.37% in language detection across various Indian languages within the IndicCorp dataset. This performance underscores the system’s effectiveness in classifying languages like Hindi, Bengali, Tamil, and others, highlighting its potential for broader NLP applications and multilingual content processing. The necessity for multilingual classification arises from the need to support diverse linguistic communities and facilitate seamless communication. Accurate language detection enables various downstream applications such as machine translation, sentiment analysis, and information retrieval, enhancing user experience and accessibility. This study emphasizes the practical application of Naive Bayes for language detection and underscores the value of the IndicCorp dataset in developing language processing tools tailored to India’s linguistic landscape. The achieved accuracy demonstrates our approach’s robustness in handling diverse linguistic characteristics and showcases its potential impact on communication and information retrieval in multilingual contexts.