Distance-Based Compression Method for Large Language Models

The computational cost of the Transformer architecture is highly dependent on the length of the input sequence, with a computational complexity of O(n2) due to the self-attention mechanism. As a result, Transformer-based models, such as Large Language Models, incur significant computational and storage overhead when processing tasks involving long input sequences. To mitigate these challenges, we propose a compression method that allows users to manually adjust the trade-off between compression efficiency and model performance. The method employs a trainable model to minimize information loss, ensuring that the impact on accuracy remains minimal. The method demonstrated an accuracy degradation within acceptable limits on LongBench v2.