500xCompressor: Generalized Prompt Compression for Large Language Models

Prompt compression is important for large language models (LLMs) to increase inference speed, reduce costs, and improve user experience.However, current methods face challenges such as low compression ratios and potential training-test overlap during evaluation.To address these issues, we propose 500xCompressor, a method that compresses natural language contexts into a minimum of one special token and demonstrates strong generalization ability.The 500xCompressor introduces approximately 0.3% additional parameters and achieves compression ratios ranging from 6x to 500x, achieving 27-90% reduction in calculations and 55-83% memory savings when generating 100-400 tokens for new and reused prompts at 500x compression, while retaining 70-74% (F1) and 77-84% (Exact Match) of the LLM capabilities compared to using non-compressed prompts.It is designed to compress any text, answer various types of questions, and can be utilized by the original LLM without requiring fine-tuning.Initially, 500xCompressor was pretrained on the Arx-ivCorpus, followed by fine-tuning on the Arx-ivQA dataset, and subsequently evaluated on strictly unseen and cross-domain question answering (QA) datasets.This study shows that KV values outperform embeddings in preserving information at high compression ratios.The highly compressive nature of natural language prompts, even for detailed information, suggests potential for future applications and the development of a new LLM language. 1