Context Embeddings for Efficient Answer Generation in Retrieval-Augmented Generation

<i>Retrieval-Augmented Generation (RAG)</i> allows overcoming the limited knowledge of LLMs by extending the input with external information. As a consequence, the contextual inputs to the model become much longer slowing down decoding time affecting the time a user has to wait for an answer. We address this challenge by presenting <i>COCOM,</i> an effective context compression method, reducing long contexts to only a handful of <i>Context Embeddings,</i> speeding up the generation time by a large margin. Our method allows for different compression rates, trading off decoding time for answer quality. Compared to earlier methods, <i>COCOM</i> allows for handling multiple contexts more effectively, significantly reducing decoding time for long inputs. Our method demonstrates an inference speed-up of up to 5.69 times while achieving higher performance compared to existing efficient context compression methods