Distance aware text optimizer for multimodal prompt learning

Multimodal prompt learning has shown remarkable generalization abilities in pretrained vision-language models, such as CLIP, enabling adaptation to various downstream tasks. Building on CoOp, previous works combine learnable text tokens with class labels to acquire domain-specific textual knowledge. However, this approach struggles with poor generalization to unseen classes, as much of the general knowledge acquired during pretraining is forgotten. To address this issue, we propose the Distance-Aware Text Optimizer (DATO), which mitigates the forgetting problem in learnable prompts. DATO introduces a distance-based penalty on text features, guiding the learning of prompts by enforcing proximity to the original prompt features. This penalty increases as the learned text features deviate further from the original ones, balancing against classification loss. Our experiments demonstrate that the proposed Distance-Aware Text Optimizer is an effective multimodal prompt learning optimization method, achieving 1.28% improvement over MaPLe.