Extending Koshur Stopwords List using Threshold-based Approach

The Kashmiri language, known as ‘Koshur by its native speakers is written in three different scripts: Sharda, Perso-Arabic, and Kashmiri-Devanagari. Stopwords removal is an essential pre-processing technique in Natural Language Processing (NLP) that increases the speed of processing. The authors had already published a list of 279 Koshur stopwords that are bridging the gap. This research paper authors experimented with the threshold value from 7 till 13. This not only contributed to the existing list of stopwords but also showed the importance of the threshold values and how little change in the value of the threshold changes the results and affects the experiment. The authors used a dataset of 352 poems, 16 stories, and 18 Articles/Blogs for the generation of tokens. A total 23671 number of tokens were extracted and used for the experiment. The experiment not only showed the importance of using threshold but also contributed to adding 32 new stopwords to the previous list, which were checked and validated by the language experts with 100% accuracy. The words were then translated into Hindi and English, plus transliteration was done in the Perso-Arabic script, another script used for the Koshur language. This research marks the addition to the previous work done by the authors.