Measuring Context Compression in a Long-Term Memory Layer: Token Savings, Accuracy, and What We Did Not Confirm
We report measurement results from running a long-term memory middleware layer (Belcore) against the LongMemEval_S benchmark. The primary finding is a 93.9% reduction in context tokens per call (from 109,079 tokens for a full transcript to 6,671 tokens for an assembled memory context), with a billed input of approximately 9,908 tokens per call on a 27-question subset. End-to-end QA accuracy in a lab harness reached 92.0% (459/499, 95% CI 89.3–94.1) using gpt-5 as the answer model. We also report a negative result: a multi-pass retrieval gate did not confirm a net improvement on a random sample (+2 within a ±5.2 noise bar, McNemar p = 0.625), and cost approximately 2.9× the input tokens. We describe what the benchmark measures, what it does not, and the limits that readers should apply before generalizing these numbers.
Authors
- Evelyn Yu
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24276754
- Primary Topic
- Topic Modeling
- Type
- preprint