Towards Informative Text Compression: A Linguistically guided Self-Information Method

Though Large Language Models (LLMs) have proven effective in the realm of Natural Language Processing (NLP), their application on long text has been hindered by huge computation cost. To meet this challenge, text compression reduce the input length by selectively removing text units, serving as a promising solution. However, current text compression methods suffer from ignoring linguistic information, resulting in missing linguistically important text units and degraded downstream task performance. To solve this problem, we introduce a novel text compression method based on linguistically guided self-information, where the self-information of text units are rescaled as importance score by linguistic features. Then, the text units with low importance score are removed, allowing LLMs to process the most linguistically and semantically important sequence. Experimental results on Natural Questions and Trivia QA datasets show our method outperforms competitive Selective Context and LLMLingua methods in terms of EM scores, balancing compression and retention well with strong performance and adaptability.