MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

Long Context Understanding (LCU) is a critical area for exploration in current large language models (LLMs).However, due to the inherently lengthy nature of long-text data, existing LCU benchmarks for LLMs often result in prohibitively high evaluation costs, like testing time and inference expenses.Through extensive experimentation, we discover that existing LCU benchmarks exhibit significant redundancy, which means the inefficiency in evaluation.In this paper, we propose a concise data compression method tailored for longtext data with sparse information characteristics.By pruning the well-known LCU benchmark LongBench, we create MiniLongBench.This benchmark includes only 237 test samples across six major task categories and 21 distinct tasks.Through empirical analysis of over 60 LLMs, MiniLongBench achieves an average evaluation cost reduced to only 4.5% of the original while maintaining an average rank correlation coefficient of 0.97 with Long-Bench results.Therefore, our MiniLongBench, as a low-cost benchmark, holds great potential to substantially drive future research into the LCU capabilities of LLMs.See Github for our code, data and tutorial.Summarization GovReport 3-1 Rouge-L English 8,734 7592 12 ( 94%) QMSum 3-2 Rouge-L English 10,614 8,253 6 ( 97%) MultiNews 3-3 Rouge-L English 2,113 1,785 11 ( 95%) VCSUM 3-4 Rouge-L Chinese 15,380 10,400 6 ( 97%) Few-shot Learning TREC 4-1 Acc.(CLS) English 5,177 6,077 8 ( 96%) TriviaQA 4-2 F1 English 8,209 9,719 12 ( 94%) SAMSum 4-3 Rouge-L English 6,258 5,974 15 ( 93%) LSHT 4-4 Acc.(CLS) Chinese 22,337 22,759 8 ( 96%) Synthetic Task PassageCount 5-1 Acc.(EM) English 11,414 10,627 4 ( 98%) PassageRetrieval-en 5-2 Acc.(EM) English 9,289 9,394 15 ( 93%) PassageRetrieval-zh 5-3 Acc.(EM) Chinese 6,745 6,684 15 ( 93%) Code Completion LCC 6-1 Edit Sim Python/C#/Java 1,235 1,187 26 ( 95%) RepoBench-P 6-2 Edit Sim Python/Jave 4,206 3,723 23 ( 95%)