Recovering Cache Reuse in Hybrid LLMs Through Checkpoint Alignment

Models that combine attention and recurrent layers can reduce inference memory requirements. This hybrid cache approach needs saved checkpoints to reuse earlier computation. In contrast, standard attention caches retain keys and values for individual tokens. We identified a mismatch between how coding agent harnesses replace task state and where hybrid caches save checkpoints, causing unchanged conversation to be processed again. We examined 17 harnesses and measured this behavior through deployed providers, vLLM and llama.cpp. Our approach recovers cache reuse while preserving compact replacement of task state. A controlled comparison on nine traces shows the tradeoff: retaining or repeatedly fetching state enlarged prompts, while an earlier anchoring method reduced hybrid recomputation by 70% versus unprotected replacement. Client side padding reduced median recomputation by 52% across eight deployed sessions. A server side checkpoint hint leaves prompt tokens unchanged and adds no model request. On the hybrid model Qwen3.5-4B, it reduced recomputation by 20% and time to first token by 16% in serial replay. With four concurrent sessions, it reduced mean block 95th percentile time to first token by 41%. Full response time did not detectably improve in serial replay. Output differences do not establish a change in task quality. These results show how checkpoint placement can recover reuse without accumulating old task snapshots. Author preprint. This manuscript is under peer review at MLSys 2027 and has not been accepted for publication.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24262436
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Recovering Cache Reuse in Hybrid LLMs Through Checkpoint Alignment

Alexander Giannini, Eric Allione
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

Recovering Cache Reuse in Hybrid LLMs Through Checkpoint Alignment

Alexander Giannini, Eric Allione
preprint en

Abstract

Models that combine attention and recurrent layers can reduce inference memory requirements. This hybrid cache approach needs saved checkpoints to reuse earlier computation. In contrast, standard attention caches retain keys and values for individual tokens. We identified a mismatch between how coding agent harnesses replace task state and where hybrid caches save checkpoints, causing unchanged conversation to be processed again. We examined 17 harnesses and measured this behavior through deployed providers, vLLM and llama.cpp. Our approach recovers cache reuse while preserving compact replacement of task state. A controlled comparison on nine traces shows the tradeoff: retaining or repeatedly fetching state enlarged prompts, while an earlier anchoring method reduced hybrid recomputation by 70% versus unprotected replacement. Client side padding reduced median recomputation by 52% across eight deployed sessions. A server side checkpoint hint leaves prompt tokens unchanged and adds no model request. On the hybrid model Qwen3.5-4B, it reduced recomputation by 20% and time to first token by 16% in serial replay. With four concurrent sessions, it reduced mean block 95th percentile time to first token by 41%. Full response time did not detectably improve in serial replay. Output differences do not establish a change in task quality. These results show how checkpoint placement can recover reuse without accumulating old task snapshots. Author preprint. This manuscript is under peer review at MLSys 2027 and has not been accepted for publication.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon