Multi-Scale Feature Fusion Using Matrix Decomposition for Enhanced Transformer Performance on Small-Scale Datasets

The Transformer model has achieved notable success in various sequence generation tasks such as machine translation and generative text summarization. However, its performance remains suboptimal in low-resource scenarios. This limitation stems from insufficient learning of model parameters and the reliance on a single-scale, global modeling mechanism. Inspired by multi-scale approaches in computer vision and the dot-product mechanism in Transformer attention, we propose a simple yet parameter-efficient method to enhance the model’s learning capacity without increasing its parameter count. The method involves constructing multi-scale features with diverse attributes from the model’s hidden dimension through matrix decomposition-based sampling, followed by the fusion of attention weights generated from these multi-scale features. This fusion results in a more robust data representation, allowing the model to learn from the data more effectively. Experimental results show that our method outperforms baseline approaches by an average of 0.45 BLEU points across six machine translation tasks and achieves a 0.55 ROUGE point improvement on the CNN-DailyMail summarization test set.