Redefining Bangla Text Processing with a New Stemming Methodology

Stemming is a vital preprocessing step in natural language processing (NLP) that reduces words to their root forms, enabling efficient text analysis. For Bangla, a morphologically rich language, stemming is particularly challenging due to complex noun and verb inflections. Existing methods often fail to handle compound words, irregular forms and over-or under-stemming issues effectively. This paper introduces a hybrid Bangla stemming technique combining numerical mapping, Part-of-Speech (POS) tagging, rule-based methods and the Levenshtein Distance algorithm. Leveraging an 130,000-word dictionary and a multi-layer fallback mechanism, the proposed system improves accuracy and flexibility. Experimental results on 1,000 unique words show 84.6% accuracy and an 82% F1-score, outperforming other Bangla stemmers. The approach demonstrates significant potential for enhancing NLP tasks like text classification and information retrieval. Future research will address limitations such as named entity processing and dictionary expansion, aiming for even greater adaptability and efficiency.