LLM Shots: Best Fired at System or User Prompts?

Over the past few years, the range of use cases involving Large Language Models (LLMs) has grown dramatically. Alongside this growth, many techniques have been established by the community to boost LLM performance. Among them, relying on the fact that LLMs excel at reproducing behaviors, practitioners have been charging their prompts with examples (a.k.a. shots) to guide or orientate the LLMs towards the correct directions given their main instruction. More recently, LLMs have evolved, allowing users to define overall roles by offering two inputs: system and user messages, based on the assumption that -in a sense- system instructions would be dedicated to admin/designer of chatbot interfaces. In such a setting: what is the best place to give example to the LLM so to improve its performances? In this study, we address this research question by systematically trying different shooting combinations with different popular benchmarks across a large set of LLMs. Our experiments show that it tends to be more beneficial to guide the LLMs through their system prompting mechanisms, leaving only the questions into their user messages.