Buffer with Feature Tokens: Combating Confusion in Medical Image Continual Learning

Due to its complexity and privacy concerns, medical data is often difficult to collect fully at once. New data emerges with the discovery of new diseases and advances in medical technology, but privacy concerns limit its storage. When deep learning models require training on datasets of new diseases, they often suffer from catastrophic forgetting, where the models’ performance on previously learned data degrades significantly. Recently, pre-trained continual learning methods have been proposed. Unlike earlier continual learning methods, these new approaches do not require learning excessive amounts of new knowledge, furthermore, they can still achieve good performance even when the pre-trained feature extractor is completely frozen. For these approaches, the performance degradation is not primarily due to catastrophic forgetting, but rather confusion. To combat confusion, we propose a method using a buffer with feature tokens (BWFT) which is cheaper and safer than directly storing the original data. Additionally, we propose a feature merge method to further reduce storage and computation costs. In the class-incremental scenario, our method outperforms state-of-the-art methods on multiple medical datasets as well as natural image datasets. Code is available at https://github.com/1345475677/BWFT.