Text Information Retrieval in Tetun

Ensuring access to information in all languages is crucial for bridging disparities in communities' participation in the digital age and fostering a more inclusive and equitable society, particularly for speakers of low-resource languages (LRLs). However, enabling such access remains a significant challenge for many of these communities. Tetun, a dialect and lingua franca that became one of Timor-Leste's official languages when the country restored its independence in 2002, faces similar challenges. According to the 2015 census, Tetun is spoken by approximately 79% of the country's 1.18 million population. Despite its official status, Tetun remains underserved in language technology. Specifically, information retrieval-based solutions for Tetun do not exist, making it challenging to find relevant information on the internet and digital platforms through text-based search in the language. This work tackles these challenges by investigating retrieval strategies for text-based search that can enable the application of information retrieval (IR) techniques to develop search solutions for Tetun, with a specific focus on the ad-hoc text retrieval task. Given that language-specific algorithms, tools, and document collections for Tetun were previously unavailable, this work began by creating these foundational resources, which serve as contributions relevant to the IR and natural language processing (NLP) domains. These resources include a tokenizer, a language identification model, a stemmer, a stopword list, a document collection, a test collection, baselines for the ad-hoc text retrieval task, and a search log dataset. The contributions to IR for LRLs include: (1) A data collection pipeline tailored for LRLs to streamline the construction of textual data from the web; (2) A human-in-the-loop methodology for annotating, processing, and constructing a dataset well-suited for a variety of IR and NLP tasks; (3) A novel network-based approach for stopword detection; (4) Methodologies for developing a stemmer, designed for a language heavily influenced by loanwords, and the construction of a ground truth set for evaluating stemmer performance; (5) A detailed approach for constructing a test collection to evaluate the effectiveness of retrieval systems; (6) A methodology for establishing a robust baseline for the ad-hoc text retrieval task; and (7) Document contextualization and dual-parameter tuning strategies for hybrid text retrieval. The results from this work contribute to the development of technologies associated with the computational processing of Tetun, address gaps in its linguistic resources, and achieve impactful outcomes that elevate Tetun's status. These advancements open new opportunities for future research and innovation. Moreover, this work introduces promising methodologies that can be adapted to other languages facing similar challenges, thereby contributing to the broader advancement of IR for LRLs. Awarded by: University of Porto, Porto, Portugal on 1 September 2005. Supervised by: Sérgio Nunes. Available at: https://hdl.handle.net/10216/169208.