Dynamic Prompt Selection for Resource-Efficient Continual Learning in Panoptic Segmentation

Modern computer vision systems face the ongoing challenge of learning new visual categories while retaining existing knowledge, particularly in comprehensive scene understanding tasks like panoptic segmentation. While visual prompt tuning (VPT) has recently shown promise in addressing this challenge, the growing collection of prompts across learning steps can lead to computational inefficiency. We introduce ECLIPSE-R, an enhancement to the visual prompt tuning (VPT) based ECLIPSE framework by introducing a dynamic prompt selection mechanism. At its core a lightweight router network that learns to identify and activate the most relevant prompts for each input image from the accumulated prompt pool across learning steps. The router network is trained end-to-end alongside the segmentation model using a combination of task-specific objectives and sparsity-inducing regularization to promote efficient prompt selection. Experimental results on the ADE20K dataset demonstrate that ECLIPSE-R achieves state-of-the-art performance in continual panoptic segmentation while maintaining computational efficiency. Our approach particularly excels in learning new classes without compromising performance on previously learned ones.