Learning how to transfer: A lifelong domain knowledge distillation framework for continual MRC

Machine Reading Comprehension (MRC) has attracted wide attention in recent years. It can reflect how well a machine understands human language. Benefitting from the increasing large-scale benchmark and pre-trained language models, a lot of MRC models have achieved remarkable success and even exceeded human performance. However, real-world MRC systems need incrementally learn from a continuous data stream across time without accessing the previously seen data, called Continual MRC system. It is a great challenge to learn a new domain incrementally without catastrophically forgetting previous knowledge. In this paper, MK-MRC (an extension of MA-MRC), a continual MRC framework with uncertainty-aware fixed M emory and lifelong domain K nowledge distillation, is proposed. MK-MRC is a memory replaying based method, in which a fixed-size memory buffer stores a small number of samples in previous domain data along with an uncertainty-aware updating strategy when new domain data arrives. For incremental learning, MK-MRC fully uses the domain adaptation and transfer relationship between memory and new domain data through several domain knowledge distillation strategies. Compared with MA-MRC, MK-MRC additionally introduces more strategies to strengthen the ability of continual learning, such as data augmentation and special task-related knowledge distillation. Experimental results show that MK-MRC yields consistent improvement compared with strong baselines and has a substantial incremental learning ability without catastrophically forgetting under four continual span-extractive and multiple-choice MRC settings. • We proposer a continual learning framework for span-extractive and multiple-choice MRC task, MK-MRC, which incrementally learns when a new domain comes. • We propose task-related knowledge distillation to transfer the knowledge about the characters of MRC task, and general-related distillation to implement normal domain adaptation. • We apply uncertainty-aware Memory and domain Knowledge distillation to MRC model. • We evaluate MK-MRC in four different continual settings. Experimental results show that MK-MRC can improve incremental learning ability without largely forgetting compared with strong baselines.