TDPO: two-stage differentiated prompt optimization for image classification

Prompt engineering has emerged as an effective paradigm for adapting vision-language models (VLMs) to image classification. Although large language models (LLMs) could generate rich visual descriptions, their hallucination often produces inaccurate or weakly discriminative class-specific prompts. Moreover, existing prompt optimization methods uniformly optimize all classes, overlooking inter-class heterogeneity. To address these limitations, we propose a two-stage differentiated prompt optimization (TDPO) that refines prompts from task-specific templates to class-specific descriptions. In the first stage, TDPO optimizes templates via roulette-wheel selection to balance efficiency and diversity. In the second stage, salient classes are sampled via a class group sampling strategy, and prompts are optimized using hybrid breeding optimization algorithm (HBO). This cooperative algorithm assigns class-specific prompts to three evolutionary lines based on inter-class heterogeneity, enabling the efficient discovery of discriminative class-specific prompts while preserving stable prompts for well-performing classes. In the setting of challenging one-shot image classification, extensive experiments on seven image classification datasets demonstrate that the proposed TDPO effectively improves classification accuracy, particularly on the Flowers102 and DTD datasets with gains of 2.9% and 1.7% over the previous state of the art. This work provides a promising direction for prompt optimization in vision-language applications.