Binary mask tuning on gradient: Towards multi-data question answering

Pretrained language models have been widely applied in question-answering tasks. To achieve better generalization on unseen datasets, several previous studies often trained a single model on multiple datasets. However, owing to the noise and distribution differences among various datasets, the model tends to excessively adjust its weights during training on multiple datasets. This deviation from the initial pretraining state results in excellent performance on specific data but overfitting on other datasets. Ultimately, the model loses its ability to generalize to new data. In this paper, from the perspective of imposing constraints on model weights, we propose a novel fine-tuning method, binary mask tuning (BMT). We employ a carefully designed binary mask vector that is closely related to the data distribution to mask the gradient generated by backpropagation , achieving precise alignment of the subspace parameters required to fit the data from a huge parameter space. This approach aimed to enhance the adaptability of the model to the data distribution and improve parameter efficiency via more targeted fine-tuning. Our experiments demonstrate that BMT is not only effective in mitigating the tendency of the model to excessively adjust its weights but also in better capturing cross-dataset regularities and dataset-specific attributes in question-answering tasks across different datasets. • We propose Binary Mask Tuning, a new fine-tuning method. • We design a binary mask vector associated with data distribution. • Our method can effectively capture cross-dataset regularities and attributes.