Advancing NLP for Shahmukhi Punjabi: Word Embedding and Text Classification with a Novel Dataset

The Punjabi language occupies a large pool in today’s era; millions speak it. Only in Pakistan, 80 million people speak the Punjabi language in the province of Punjab. However, with this Big Market Cap, there has yet to be any proper research available. This research focuses on the Punjabi Language, especially the Shahmukhi Punjabi language, famous in Pakistan and Asia. However, it needs to be given more attention in the existing research. There needs to be an adequately supervised established dataset available with large data. Till now, there has yet to be any proper research on Word Embedding and Classification. This paper introduced the crafted dataset for the Shahmukhi Punjabi language dataset. It is also based on advanced NLP techniques like Word2Vec and SDfasttext for Word Embedding to capture the semantic relation within the language. In addition, we investigated the applications of six distinct classification models to analyze four different categories: News, Ghazal, Dohra, and Poetry. The notable success of the Naive Bayes classifier with other classification models lays the groundwork for future research and applications in natural language processing for the Punjabi language. The study encourages further exploration and the development of tailored solutions to meet the linguistic diversity in digital environments and apply deep learning models.