A Preliminary Study On End-to-End Multimodal Subtitle Recognition for Taiwanese TV Programs

Taigi or Taiwanese, once widely spoken throughout Taiwan, has experienced a significant decline and is now considered endangered. This research aims to develop an end-to-end subtitles recognition model that can annotate Taiwanese TV programs to automatically create large-scale Taiwanese au-diovisual datasets. This effort supports the development of advanced multi-modal AI models and the preservation of the Taiwanese language. Our approach involves developing an “End-to-End Multimodal Subtitles Recognition“ system for Taiwanese TV, which requires models to generate subtitles by integrating simultaneous textual and visual information. Further-more, several large language model-based decoders, such as BART, GPT-2, and LLaMA2, were fine-tuned to improve performance. To validate our approach, a multimodal audi-ovisual subtitles dataset comprising 238,713 video segments from PTS-Taigi [1] TV programs was created. Our subtitles recognition model achieved the CER (Character Error Rate) of 9.37% for bilingual subtitles in Chinese and Taiwanese. Specifically, the CER for Taiwanese subtitles is 4.43%, while the CER for Chinese subtitles is 16.54%.