Can Language Models Serve as Analogy Annotators?

Conceptual abstraction and analogy-making are crucial for human learning, reasoning, and adapting to unfamiliar domains.Recently, large language models (LLMs) have made the synthesis of analogical data possible, which, however, still heavily relies on extensive human efforts to be annotated.This paper empirically examines the LLMs' capability to annotate story-level analogical data.Specifically, we propose a novel multi-stage progressive reasoning prompt framework A3E (Automated Analogy Annotation Expert), which is based on the structure mapping theory from cognitive psychology and efficiently annotates candidate story pairs across six fine-grained categories.We use A3E to evaluate how well the state-of-the-art LLMs can serve as analogy annotators.Experimental results demonstrate that our proposed A3E achieves an average performance gain of + 73% across a range of prompting baselines and base LLMs.The code and data can be available at https://github.com/zhangxjohn/A3E.