Sizing GPU Memory for Large Language Model Inference: A Practical Model of Weights, KV Cache, and Runtime Overhead

Technical report (preprint) — NOT peer-reviewed. This report presents calculations only; no new hardware benchmarks were conducted. Abstract. This report presents a practical model for sizing GPU memory (VRAM) for large language model inference, accounting for model weights, the key-value (KV) cache, and runtime overhead. For an 8B-parameter model in BF16 precision, the model estimates 20.46 GB of VRAM for a single sequence with 16,384 tokens, and 22.60 GB for 8 concurrent users with 4,096 tokens each. These figures are estimates intended for capacity planning only; actual memory usage varies with implementation, framework, and hardware.