Mitigating Catastrophic Forgetting in Molecular Property Prediction via Refresh Learning and Pareto Optimization

Continual Learning (CL) enables Large Language Models (LLMs) to adapt to new episodes and data streams without forgetting previously acquired knowledge, a critical requirement for dynamic fields like Molecular property Prediction (MP). LLMs, however, often face Catastrophic Forgetting (CF), where learning new information erodes prior knowledge, particularly when data distributions shift significantly between episodes, as seen in chemical, genomic, and proteomic datasets. To address CF, the existing replay-based techniques use memory buffers to store past episode data but often overlook the relationships between episodes, resulting in sub-optimal performance when revisiting earlier episodes. To this, the paper proposes a Multi-task Learning (MTL) framework that reconciles existing CL techniques into a unified hierarchical gradient aggregation framework. It builds a novel framework using the ChemBERTa model, namely MTL-PORL (Multi-task Learner-Pareto Optimized Refresh Learning), i.e., Refresh Learning (RL), inspired by neuroscience, where the brain discards outdated information to enhance retention and facilitate new learning with Pareto Optimization (PO) for MP. The hyper-gradient approach in the MTL-PORL leverages unlearning current data before relearning it, acting as a flexible plug-in that enhances existing CL methods. The MTL-PORL exhibits Anytime Average Accuracy (91.63%, 94.89%, and 92.67%), Test Accuracy (92.48%, 96.48%, and 96.86%), and Forgetting Measure (-0.0048, -0.0045, and -0.0063) on the BBBP, bitter, and sweet datasets, respectively. The comprehensive empirical analysis highlighted significant improvements in sequential learning compared to existing methods, addressing the stability-plasticity trade-off and effectiveness of RL.