When Does Keeping Agent Sessions in GPU Memory Pay Off? A Rust Serving Engine and a Replay Study on Real Agent Traces

AI agents call a language model, run a tool, and call the model again with the whole historyattached. Keeping an agent’s KV cache in GPU memory while it waits on a tool avoids recomputingthat history, but holds memory other requests could use. Prior systems (InferCept,Pensieve, Continuum, TokenCake and others) model this trade-off and evaluate it against earlierversions of vLLM. We ask how it plays out against the current prefix caches of vLLM 0.33.0 andSGLang 0.5.23. We built a serving engine in Rust (about 1,700 lines) with per-session KV retention,cross-session prefix sharing and session-level admission, and replayed 24 real coding-agentsessions (791 turns) against all three engines with the same GPU, model and KV cache size, tworuns per configuration. With a fixed 0.5-second tool wait, close to the median gap measuredfor coding agents, our engine finishes the workload 1.31x faster than the best baseline (1.45xfaster than vLLM) and returns the first token about 5x sooner. With a 5-second wait the gain is1.13x (1.24x over vLLM). With a 60-second wait, all engines finish within 2% of each other andour engine is slower to the first token, consistent with prior reports that retention stops payingoff when agents are away long. Three pinning policies we tried, each with a pass/fail criterionwritten down before seeing its results, did not beat the baselines at 60 seconds.

Authors

Publication Details

Journal
Preprints.org
Published
2026-11-13
DOI
https://doi.org/10.20944/preprints202611.1388.v1
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Does Keeping Agent Sessions in GPU Memory Pay Off? A Rust Serving Engine and a Replay Study on Real Agent Traces

Nahuel Alejandro Nucera
2 links
Preprints.org
Parallel Computing and Optimization Techniques
preprint

When Does Keeping Agent Sessions in GPU Memory Pay Off? A Rust Serving Engine and a Replay Study on Real Agent Traces

Nahuel Alejandro Nucera
preprint en

Abstract

AI agents call a language model, run a tool, and call the model again with the whole historyattached. Keeping an agent’s KV cache in GPU memory while it waits on a tool avoids recomputingthat history, but holds memory other requests could use. Prior systems (InferCept,Pensieve, Continuum, TokenCake and others) model this trade-off and evaluate it against earlierversions of vLLM. We ask how it plays out against the current prefix caches of vLLM 0.33.0 andSGLang 0.5.23. We built a serving engine in Rust (about 1,700 lines) with per-session KV retention,cross-session prefix sharing and session-level admission, and replayed 24 real coding-agentsessions (791 turns) against all three engines with the same GPU, model and KV cache size, tworuns per configuration. With a fixed 0.5-second tool wait, close to the median gap measuredfor coding agents, our engine finishes the workload 1.31x faster than the best baseline (1.45xfaster than vLLM) and returns the first token about 5x sooner. With a 5-second wait the gain is1.13x (1.24x over vLLM). With a 60-second wait, all engines finish within 2% of each other andour engine is slower to the first token, consistent with prior reports that retention stops payingoff when agents are away long. Three pinning policies we tried, each with a pass/fail criterionwritten down before seeing its results, did not beat the baselines at 60 seconds.

Preprints.org
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon