Justify Your Prompts!

Abstract When you use a large language model (LLM) in your research, you often need to formulate a prompt to elicit some relevant output from the LLM. This step is challenging since (1) LLMs are known to be brittle and their results may vary drastically between different prompts; and (2) for any given task, there are infinitely many possible prompts. Thus we end up with the following problem: if you cannot try out infinitely many prompts, how do you justify your selected prompt? This paper discusses different ways in which authors may motivate their choices, and urges authors to consider how their prompting strategy might be justified.