Issues and Challenges in Training Vector Representation for Assamese Words

Dense numeric vectors can efficiently represent the syntax and semantics of words and are useful across multiple natural language processing tasks. Assamese, a low-resource language with substantial native speakers in India, lacks high-quality word embeddings to support such tasks. To better understand their effectiveness, this study provides an intrinsic evaluation of the existing Assamese pre-trained word embeddings on a custom 600-word analogy question dataset and compares the results with those of a high-resource language. We posit that such results may be attributed to the underlying corpus. To investigate this possibility, we conduct an in-depth examination of the largest available Assamese corpus. We find that training high-quality embeddings for Assamese is hindered by its morphological complexity, limited tools, and the presence of extensive noise in the available corpora. Finally, we propose techniques to mitigate these issues and improve the quality of the embeddings.