APRA: Adaptive Prompt Refinement with Early Stopping for Efficient Automatic Prompt Optimization in Generative AI

The automatic prompt optimization (APO) research area moved beyond prompt editing into independent research direction in which the language models generate the criticism, candidates, or reflections to optimize the prompt to maximize the task performance. Current approaches include such techniques as textual gradients and beam search, evolutionary and reflective optimizers. Though these algorithms optimize prompt without changing parameters of the model, their search strategy is expensive in terms of computation and is not task-dependent. This work suggests a light-weight Adaptive Prompt Refinement Algorithm (APRA), which implements three components – best prompt tracking, improvement threshold, and patience-based stopping criteria into the single-candidate refinement procedure. The research question is whether the task-dependent adaptive allocation of inference budget is possible based on feedback from the development set. We state the problem, provide the formal cost modeling, compare APRA with other APO methods and design reproducible small-scale experiment based on the classification task. The particular attention will be paid to the validity threats since one-hour pilot is not enough to confirm superiority of our approach over other APO approaches. No experimental results were fabricated in this manuscript: the tables are designed to be filled in based on the actual run of the experiments. Thus, the novelty of this manuscript lies in the proposed optimization policy rather than superiority over other APO systems.