Tampons de Répétition Distribués pour passer l’Apprentissage Continu à l’Échelle

Deep Learning (DL) emerged as a way to extract valuable information from ever-growing volumes of data. However, when trained on sequential tasks i.e., without full access to the dataset at the beginning of the training, typical Deep Neural Networks (DNNs) suffer from catastrophic forgetting, a phenomenon causing them to reinforce new patterns at the expense of previously acquired knowledge. This limitation prevents updating models incrementally, which is problematic in many real-life scenarios where the aforementioned datasets are replaced by data streams generated over time. It is unfeasible to train models from scratch every time new samples are being ingested either, as this would lead to prohibitive time and/or resource constraints.In this dissertation, we present techniques based on rehearsal to achieve Continual Learning at scale. Rehearsal-based approaches utilize representative samples previously encountered during training to augment future minibatches with. The key novelty we address is how to adopt rehearsal in the context of data-parallel training, which is one of the main techniques to achieve training scalability on HPC systems. We design a distributed rehearsal buffer that leverages parallelization techniques, enabling us to improve model performance without increasing training time.