CUDA device memory is invisible to Linux memory accounting on the NVIDIA GB10: two freezes of a DGX Spark, a five-minute safe test, and a two-layer guard
The NVIDIA DGX Spark (GB10) gives its CPU and GPU one pool of 121.7 GiB of LPDDR5X memory. When a GPU workload exhausts that pool, the machine does not fail one process: on our unit it stopped serving SSH, stopped running scheduled jobs, and needed a power cycle, twice (2026-09-03 and 2026-10-27), on driver 580.173.02, the build after the updates NVIDIA said on its forum stop such freezes. We measured where an ordinary CUDA allocation shows up in Linux's own accounting. An 8 GiB cudaMalloc made inside a Docker container limited to 4 GiB succeeds; it raises the process's resident set by 0.11 GiB and the container's memory cgroup by 0.10 GiB, leaves the kernel's out-of-memory score unchanged, and lowers the system's MemAvailable by 8.74 GiB. Host, pinned and CUDA-managed allocations of 2-3 GiB are, by contrast, charged in full, and a managed allocation over the container limit is killed by the cgroup's out-of-memory handler. This explains both incidents. In the second, the kernel's global out-of-memory killer did run, 25 minutes into the stall, and killed a 1.1 MiB dashboard process: the 146 processes in its table held 0.55 GiB resident between them, while about 120 GiB of the pool sat in device allocations it cannot attribute. Container memory limits, the out-of-memory killer and cgroup-based tools therefore cannot protect a GB10 from a CUDA workload. Two userspace measures do: a cap inside the CUDA allocator, which turns an over-budget request into a catchable error, and an external watcher on MemAvailable at 0.5 s that kills the workload above a safety floor. With both, the workload that froze the machine ran 200 training steps at a peak of 15.4 GiB. The test takes five minutes, never approaches exhaustion, and is published with this note (source-and-test-data.tar.gz: LaTeX source, measurement scripts, raw run logs and redacted kernel logs).
Authors
- Mark Hutto (ORCID: https://orcid.org/0009-0005-7841-6262)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24239250
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint