Optimizing Byte Pair Encoding Tokenization for South African Languages

In this paper, we introduce an Optimized Byte Pair Encoding (OBPE) tokenizer where the algorithm is optimized for the South African languages, including Sesotho, Setswana, Xhosa, Xitsonga, and Zulu. Our approach is tailored to handle the characteristics of multilingual datasets and language complexity, especially regarding the morphological richness of South African Languages. Firstly, we incorporate language-specific tokens to facilitate learning of language-specific context and differentiation when training the algorithm on multilingual corpora. Secondly, we modify the existing BPE algorithm to add common words derived from the multilingual corpus to the initial vocabulary, which helps to reach convergence faster and possibly get more meaningful or accurate tokenization. Thirdly, we introduce a common words parameter that allows the user to specify the number of common words derived from the corpus to be used for the initial vocabulary of the algorithm. Our OBPE achieved accuracy of 96% compared to the traditional BPE (87-88%).