9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices

Large Language Model inference on memory-constrained devices suffers from delay due to thrashing. Since models with billions of parameters often cross the memory limit, it becomes imperative to keep the weights on disk and fetch them on demand. As a result, the inference procedure faces delay due to the constant swapping in and out of model parameters between disk and memory. In this paper, we establish this problem by comparing the existing methods in the literature. We also propose a solution to this problem, which is an efficient eviction and prefetching scheme.

Authors

Publication Details

Journal
DigitalCommons - Kennesaw State University (Kennesaw State University)
Published
2026-11-20
Primary Topic
Parallel Computing and Optimization Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices

ARNAB JOY
DigitalCommons - Kennesaw State University (Kennesaw State University)
Parallel Computing and Optimization Techniques
article

9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices

ARNAB JOY
article en

Abstract

Large Language Model inference on memory-constrained devices suffers from delay due to thrashing. Since models with billions of parameters often cross the memory limit, it becomes imperative to keep the weights on disk and fetch them on demand. As a result, the inference procedure faces delay due to the constant swapping in and out of model parameters between disk and memory. In this paper, we establish this problem by comparing the existing methods in the literature. We also propose a solution to this problem, which is an efficient eviction and prefetching scheme.

DigitalCommons - Kennesaw State University (Kennesaw State University)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
9253-GRP Efficient Memory Management for LLM Inference on Constrained Memory Devices — ARNAB JOY · DigitalCommons - Kennesaw State University (Kennesaw State University) (2026) | TGRS Research Map | TGRS