Rehearsal-based continual learning for robust ovarian cancer subtype classification under catastrophic forgetting

Ovarian cancer encompasses multiple molecular subtypes, each necessitating distinct diagnostic approaches and tailored therapeutic strategies. Achieving accurate and adaptive classification of these subtypes remains a significant challenge, particularly in clinical environments where data distributions evolve over time. To address this issue, this study proposes a novel continual deep learning framework designed to mitigate catastrophic forgetting, a key limitation of incremental learning models. The proposed framework employs a Vision Transformer–based architecture coupled with a task-specific classification head and rehearsal-based memory buffers, enabling the retention of representative knowledge from previously learned samples while facilitating effective adaptation to newly acquired data. Extensive experimental evaluations conducted on a large-scale ovarian cancer dataset demonstrate the robustness and effectiveness of the proposed approach. The model achieved a validation accuracy of 96.2% and a test accuracy of 95.4% across all evaluated ovarian cancer subtypes. Furthermore, consistent and balanced performance was observed, with F1-scores exceeding 0.92 for all subtypes, and classification accuracy of 99% for the clear cell carcinoma subtype. These findings highlight the potential of the proposed continual learning framework to provide reliable, adaptive, and clinically meaningful ovarian cancer subtype classification in real-world diagnostic settings.