Studying Forgetting in Faster R-CNN for Online Object Detection: Analysis Scenarios, Localization in the Architecture, and Mitigation

Online Object Detection (OOD) requires learning new object categories from a stream of images, similar to an agent exploring new environments. In this context, the widely used architecture Faster R-CNN (Region Convolutional Neural Network) faces catastrophic forgetting: the acquisition of new knowledge leads to the loss of previously learned information. In this paper, we investigate the learning and forgetting mechanisms of Faster R-CNN in OOD through three main contributions. First, we observe that the forgetting curves of the Faster R-CNN exhibit patterns similar to those described in human memory studies by Hermann Ebbinghaus: knowledge is lost exponentially over time and recall improves knowledge retention. Second, we present a new methodology to analyse the Faster R-CNN architecture and quantify forgetting across the Faster R-CNN components. We show that forgetting is mainly localised in the Softmax classification layer. Finally, we propose a new training strategy for OOD called Configurable Recall (CR). CR performs recalls on old data using images stored in a memory buffer with variable frequency and recall length to ensure efficient learning. CR also masks the logits of old objects in the softmax classification layer to mitigate forgetting. We evaluate our strategy against state-of-the-art methods on three OOD benchmarks. We analyse the effectiveness of different types of recall in mitigating forgetting and show that CR outperforms existing methods.