Using Semantically Unrelated and Opposite Terms for In-Context Learning: A Case Study in Identifying Political Aversion in Tweets

We investigate how semantic priors embedded in generative large language models (LLMs) interact with concept definitions in prompts, using political aversion detection as a case study.Through systematic variations in the wording of the definition of political aversion-replacing key words with opposite terms, semantically unrelated terms, or deliberately nonsensical strings-we examine how models process and apply these manipulated definitions in classification tasks.Our results show that certain LLMs maintain consistent performance across different prompt configurations, regardless of which terms are used or whether examples are included.Strong classification performances even with nonsensical definitions suggest these models may sometimes rely more on patterns in target content than definitions given in prompts.These findings challenge conventional assumptions about prompt engineering and raise important questions about how LLMs utilize information in prompts for classification decisions, while underscoring the need for careful validation when applying these methods to social and political science measurement tasks.