VAP3: Variation-Aware Prompt Performance Prediction

Large Language Models (LLMs) exhibit strong capabilities across various Information Retrieval (IR) and natural language processing tasks. However, they are highly sensitive to prompt variations, where slight rephrasings can significantly alter responses, leading to inconsistent or incorrect outputs. This variability poses challenges for response reliability in real-world applications. Inspired by Query Performance Prediction (QPP) in IR, we focus on Prompt Performance Prediction (PPP), which estimates whether an LLM will generate a correct response for a given prompt before execution. We propose VAP3 (Variation-Aware Prompt Performance Prediction), a novel pre-generation PPP approach that integrates prompt variations with adversarial training to enhance robustness against trivial modifications and better capture prompt sensitivity. Evaluating VAP3 against LLM-based self-evaluation, QPP-inspired baselines, and supervised classification models on the PromptSET-HotpotQA and PromptSET-TriviaQA datasets, we demonstrate that VAP3 consistently outperforms all baselines, achieving stable and reliable performance across datasets.