When to Inject? Investigating Different Metrics for Conditional Context Injection in LLMs

LLMs have become more and more integrated into larger software systems, in which they are provided with additional information to answer user queries. While this improves answer quality in many cases, it comes at an increased cost in tokens and thus computational load and electricity consumption. In this paper, we investigate low compute cost mechanisms which can help us predict whether an LLM would profit from a costly token injection.