CUDA device memory is invisible to Linux memory accounting on the NVIDIA GB10: two freezes of a DGX Spark, a five-minute safe test, and a two-layer guard

The NVIDIA DGX Spark (GB10) gives its CPU and GPU one pool of 121.7 GiB of LPDDR5X memory. When a GPU workload exhausts that pool, the machine does not fail one process: on our unit it stopped serving SSH, stopped running scheduled jobs, and needed a power cycle, twice (2026-09-03 and 2026-10-27), on driver 580.173.02, the build after the updates NVIDIA said on its forum stop such freezes. We measured where an ordinary CUDA allocation shows up in Linux's own accounting. An 8 GiB cudaMalloc made inside a Docker container limited to 4 GiB succeeds; it raises the process's resident set by 0.11 GiB and the container's memory cgroup by 0.10 GiB, leaves the kernel's out-of-memory score unchanged, and lowers the system's MemAvailable by 8.74 GiB. Host, pinned and CUDA-managed allocations of 2-3 GiB are, by contrast, charged in full, and a managed allocation over the container limit is killed by the cgroup's out-of-memory handler. This explains both incidents. In the second, the kernel's global out-of-memory killer did run, 25 minutes into the stall, and killed a 1.1 MiB dashboard process: the 146 processes in its table held 0.55 GiB resident between them, while about 120 GiB of the pool sat in device allocations it cannot attribute. Container memory limits, the out-of-memory killer and cgroup-based tools therefore cannot protect a GB10 from a CUDA workload. Two userspace measures do: a cap inside the CUDA allocator, which turns an over-budget request into a catchable error, and an external watcher on MemAvailable at 0.5 s that kills the workload above a safety floor. With both, the workload that froze the machine ran 200 training steps at a peak of 15.4 GiB. The test takes five minutes, never approaches exhaustion, and is published with this note (source-and-test-data.tar.gz: LaTeX source, measurement scripts, raw run logs and redacted kernel logs).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24239250
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

CUDA device memory is invisible to Linux memory accounting on the NVIDIA GB10: two freezes of a DGX Spark, a five-minute safe test, and a two-layer guard

Mark Hutto
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

CUDA device memory is invisible to Linux memory accounting on the NVIDIA GB10: two freezes of a DGX Spark, a five-minute safe test, and a two-layer guard

Mark Hutto
preprint en

Abstract

The NVIDIA DGX Spark (GB10) gives its CPU and GPU one pool of 121.7 GiB of LPDDR5X memory. When a GPU workload exhausts that pool, the machine does not fail one process: on our unit it stopped serving SSH, stopped running scheduled jobs, and needed a power cycle, twice (2026-09-03 and 2026-10-27), on driver 580.173.02, the build after the updates NVIDIA said on its forum stop such freezes. We measured where an ordinary CUDA allocation shows up in Linux's own accounting. An 8 GiB cudaMalloc made inside a Docker container limited to 4 GiB succeeds; it raises the process's resident set by 0.11 GiB and the container's memory cgroup by 0.10 GiB, leaves the kernel's out-of-memory score unchanged, and lowers the system's MemAvailable by 8.74 GiB. Host, pinned and CUDA-managed allocations of 2-3 GiB are, by contrast, charged in full, and a managed allocation over the container limit is killed by the cgroup's out-of-memory handler. This explains both incidents. In the second, the kernel's global out-of-memory killer did run, 25 minutes into the stall, and killed a 1.1 MiB dashboard process: the 146 processes in its table held 0.55 GiB resident between them, while about 120 GiB of the pool sat in device allocations it cannot attribute. Container memory limits, the out-of-memory killer and cgroup-based tools therefore cannot protect a GB10 from a CUDA workload. Two userspace measures do: a cap inside the CUDA allocator, which turns an over-budget request into a catchable error, and an external watcher on MemAvailable at 0.5 s that kills the workload above a safety floor. With both, the workload that froze the machine ran 200 training steps at a peak of 15.4 GiB. The test takes five minutes, never approaches exhaustion, and is published with this note (source-and-test-data.tar.gz: LaTeX source, measurement scripts, raw run logs and redacted kernel logs).

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon