A sentence-aligned bitext corpus for colloquial to standard Punjabi normalization for low-resource NLP applications in Gurumukhi script

In this study, a bitext dataset was developed, comprising 7109 sentence pairs that feature Punjabi colloquial expressions alongside their corresponding standard Punjabi translations. Punjabi colloquial language is characterized by code-mixing, informality, code-switching, irregularities, misspellings, misrepresentations, and the use of multiple dialects, creating a complex linguistic form. The dataset creation involved scraping digital articles, preprocessing the text, and segmenting it into coherent sentences. Each sentence was then paired with its pure Punjabi equivalent. The article summarizes the current state of Punjabi language text datasets only, as the datasets on Punjabi audio, speech, images, and handwritten Gurumukhi character recognition etcetera are out of scope of the article. The information presented is intended for a detailed understanding of the available resources. Further, to assess the quality of the bitext, three experts proficient in Gurumukhi script, evaluated the dataset. The assessment process involved three rounds of evaluation using randomly stratified samples of sentences in sizes of 25, 50, and 100 from each category (see Appendix -C). Inter-rater agreement was calculated using the Kappa Statistic, resulting in Kappa values of 0.85 between Judge 1 and Judge 2, and between Judge 2 and Judge 3, with a slightly lower agreement of 0.72 between Judge 1 and Judge 3. The overall Kappa score was 0.55, indicating a substantial level of agreement among the judges regarding the quality of the translations and usefulness for future research work in Punjabi translation and language modeling.