Dynamic Task Weighting Mechanism for a Task-Aware Approach to Mitigating Catastrophic Forgetting

Catastrophic forgetting is still a big issue in sequential learning and in particular for Natural Language Processing (NLP) models that tend to forget knowledge encoded in previous tasks when learning new targets. To do this, we present a Dynamic Task Weighting Mechanism which forms a part of the Adaptive Knowledge Consolidation (AKC) framework. Our method dynamically adjust knowledge retention to task similarity and task specific performance, while contrasted to static regularization approaches such as Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI). This mechanism is proposed that involves computing task embeddings with pre-trained models BERT and quantifying their similarity from cosine similarity. To complete the above, we compute a similarity score which is merged with normalized task specific performance metrics of accuracy, F1 score to form an importance score. The model trades adaptability in learning in order to retain previously learned knowledge by prioritizing important tasks and minimizing interference from other unrelated tasks. We show that our proposed mechanism substantially mitigates forgetting and results in accuracy improvements on extensive experiments on standard NLP benchmarks such as GLUE, AG News, and SQuAD. Among baseline methods (EWC, SI, and GEM), the model also has the highest average accuracy of 86.7% and the least amount of forgetting of 6.2%.