LatVis: Large-scale Task-specific Language Model for Low-resource Vietnamese Multi-document Summarization

The Vietnamese multi-document summarization task faces three key challenges including the long input sequence problem, human-like summary generation, and the scarcity of labeled data. Transformer-based models, enhanced by parallel computation architectures and attention mechanisms, partially mitigate the challenges posed by long input sequences. Additionally, when trained on large-scale text corpora, these models achieve impressive performance in text generation, approaching human-level performance. Furthermore, pre-training with self-supervised learning objectives is an effective strategy to compensate for the scarcity of labeled data. Based on these considerations, this article leverages large corpora of unlabeled Vietnamese text to build a large-scale pseudo-labeled multi-document dataset, which is then used to pre-train a Vietnamese task-specific language model, LatVis. Experimental results demonstrate that, even without fine-tuning, the pre-trained model achieves competitive performance compared to several previous models. After fine-tuning on approximately 300 samples, LatVis obtains notable Rouge Scores of 76.7%, 78.9%, and 73.9% for Rouge-1 F1, and 50.2%, 55.0%, and 46.7% for Rouge-2 F1 on the VMDS, ViMS, VLSP datasets, respectively. To the best of our knowledge, this is the first publicly large-scale task-specific language model pre-trained specifically for the Vietnamese multi-document summarization. This work highlights the potential of task-specific language models for advancing natural language processing for low-resource languages.