DEVELOPMENT OF A TOKENIZER FOR AN ORTHOGRAPHIC YORÙBÁ-BASED PROGRAMMING LANGUAGE TO ENHANCE INDIGENOUS LANGUAGE COMPUTING

The Yorùbá language, with its rich orthographic system, is spoken by over 100 million people worldwide. Despite its widespread use and rich linguistic heritage, the computational resources available for the Yorùbá language are limited. Current programming languages and tools are predominantly designed for English or other widely spoken languages, creating a barrier for those who are only literate in Yorùbá but interested in programming and software development. This research aims to bridge this gap by developing a robust token recognizer for a Yorùbá-based programming language that recognizes and supports the language’s modern orthography. The lexical structure of the Programming Language was specified by defining an appropriate regular grammar with the BNF notation and standard Yorùbá words used as reserved words. The Programming Language’s character set consists of the basic Yorùbá alphabet. The tokenizer system was implemented on Visual Studio IDE using C# programming language. The test data was created using an online text editor that enables the typing of Yorùbá characters with tonal marks. The developed Tokenizer can determine the different lexemes that form different tokens of the source program in the source language. Besides, it recognizes whitespace characters and detects and reports invalid characters of the source language as well as programmer-directed statements in the source program. The output produced by the system on some given test data showed its functional correctness. This initiative is crucial for advancing indigenous language computing, empowering Yorùbá speakers, and preserving the Yorùbá language through technological innovation. Furthermore, the study has the potential to contribute to driving the fourth Sustainable Development Goal of the United Nations.