Optimizing pre-training via target-aware source data selection

• Domain adaptation information gain-based target-related data selection is proposed. • Proposed DAIG improves target model accuracy via target-related data selection. • DAIG effectively gathers relevant data from source to benefit target tasks. • DAIG extracts “rough prior” from target data for source data pre-training. • DAIG-guided selection outperforms baselines in multiple experimental settings. In recent years, due to the explosive popularity of large-scale pre-trained models such as large language models, pre-training approaches that use a massive amount of source data and can be applied to various target tasks are becoming more popular. Pre-trained models allow us to learn highly accurate target models by fine-tuning them with target data, even when their volume is insufficient. However, the source data used to train a pre-training model is generally a large and miscellaneous data set obtained in the wild without being aware of the target task, and it highly possibly contains much data that does not contribute to relearning the target task. This study defines a novel paradigm as “target-aware source data selection,” which uses the source data itself instead of a pre-training model and selects source data for pre-training and aims to increase its quality, effectiveness, and robustness. Our proposal fundamentally differs from the current studies addressing the lack of target data and conventional transfer learning approaches, improving source data quality using the novel Domain Adaptation Information Gain criteria. Specifically, the target model is pre-trained while actively selecting only informative data from the source data using the “rough-prior knowledge” obtained from the target data training before the pre-training. Finally, fine-tuning the model with the target data results in a highly accurate model for the target (downstream) task. The effectiveness of our proposed paradigm has been demonstrated through multifaceted experiments using multiple pairs of target data and source data with different strengths of their relevance.