DOD: Double Online Distillation for Continual Learning

Continual Learning (CL) aims to learn incrementally from a data stream while retaining previously acquired knowledge. Many existing methods rely on offline distillation, which can propagate harmful knowledge during transfer and, when memory is limited and old and new samples are imbalanced, tends to encourage both feature recency bias and response recency bias toward new tasks. Offline distillation can further amplify these two biases. We propose Double Online Distillation (DOD), a framework for dynamic knowledge transfer and joint calibration of both biases through coupled student and teacher learning. First, the Collaborative Online Update (COU) strategy encourages mutual guidance between the student and teacher models and is designed to limit bias accumulation during knowledge transfer. Second, Online Feature Distillation (OFD) uses an attention-based similarity measure to steer the student toward high-quality features of past tasks, alleviating feature recency bias. Finally, Online Response Distillation (ORD) combines inter-instance knowledge with an adaptive perception mechanism to assess and refine teacher responses, with the aim of mitigating response recency bias. Extensive experiments on five benchmark datasets, under diverse continual learning settings and various buffer sizes, show that DOD achieves the highest mean accuracy among the compared continual learning methods in all 24 settings. Its improvements over the second-best method are statistically significant in 11 settings, mainly with small buffers or on Tiny-ImageNet; the remaining 13 settings show numerical gains without statistical significance.