Construction of Large Language Models for Taigi and Hakka Using Transfer Learning

Taigi (also known as Taiwanese Hokkien) and Hakka were once the predominant native languages in Taiwan, but they are now on the verge of extinction. To address this crisis, this study explores how to construct large language models (LLMs) for Taigi and Hakka to aid in their revitalization despite the scarcity of linguistic resources (the low-resource language issue). Our approach leverages pre-trained models from LLMs of resource-rich languages, following three key steps: (1) introducing specific characters for Taigi and Hakka into the LLM's tokenizer, enabling the model to recognize the orthography of both languages; (2) conducting continual pre-training with a small amount of Taigi and Hakka text data to familiarize the LLM with the lexical choices of these languages; and (3) generating task-specific training data via machine translation for instruction fine-tuning, allowing the LLM to perform tasks in Taigi and Hakka. Finally, we evaluated the performance of the constructed LLMs using a reading comprehension question bank (consisting of 1,000 multiple-choice questions) developed by Taiwanese and Hakka language experts. The experimental results indicate that based on LLaMA2-70B, the LLMs achieved accuracy rates of 85.6% for Taigi and 87.7% for Hakka, respectively, suggesting that the performance is approaching human-level proficiency.