Twelve Silent Defects in a From-Scratch C and CUDA Transformer Trainer: How Address-Keyed Device Caches Masqueraded as BF16 Instability

Training a transformer language model in a hand-written C99 and CUDA stack without an automatic-differentiation framework exposes a class of silent state-management failures that high-level frameworks prevent by construction. During the first pretraining attempts of a 184.7-million-parameter decoder-only transformer in ANNI (47,027 lines of repository code; no PyTorch, cuDNN, or autograd), four runs on 2026-10-25 either diverged above the uniform-guess baseline of 9.72 nats or plateaued unexpectedly. A controlled 620-step probe comparing BF16 against FP32 on the same data and seed reached a training negative log-likelihood (NLL) of 6.060 in FP32 while exploding to 27.372 in BF16, leading to the initial diagnosis that BF16 matrix products were numerically unstable at the chosen learning rate. A subsequent end-to-end audit of the CUDA backend against the CPU reference implementation revealed that the precision comparison had measured the bugs rather than the number formats. The audit uncovered twelve defects in the GPU training path. Only one defect was specific to half-precision matrix products (a cuBLASLt algorithm-timing loop that executed accumulating weight-gradient GEMMs up to 16 times in place on the first step, inflating step-1 BF16 weight gradients up to 7.3x); the four most damaging defects acted in every precision, including FP32. Worst among them, cross-entropy target arrays were uploaded through a persistent device cache keyed on the host pointer address and refreshed only when a weight-epoch counter changed. Because the host target buffer was allocated with malloc per microbatch and recycled at the same heap address, every microbatch from the second optimizer step onward was scored and back-propagated against the targets of a single fixed microbatch—yet the reported training loss continued to fall for hundreds of steps as the model memorised token positions. We describe the residency and caching architecture where these twelve defects arose, give the exact mechanism, symptom, and fix for each defect alongside four additional instrumentation and data-path bugs found later, state the six framework-free invariants and regression tests that caught them, and show that after the fixes FP32 and BF16 agree to 0.0001 nats of validation loss over 150 optimizer steps and train 185M- and 555M-parameter models cleanly across billions of tokens.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24256084
Primary Topic
Parallel Computing and Optimization Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Twelve Silent Defects in a From-Scratch C and CUDA Transformer Trainer: How Address-Keyed Device Caches Masqueraded as BF16 Instability

Vincent Abel
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
article

Twelve Silent Defects in a From-Scratch C and CUDA Transformer Trainer: How Address-Keyed Device Caches Masqueraded as BF16 Instability

Vincent Abel
article en

Abstract

Training a transformer language model in a hand-written C99 and CUDA stack without an automatic-differentiation framework exposes a class of silent state-management failures that high-level frameworks prevent by construction. During the first pretraining attempts of a 184.7-million-parameter decoder-only transformer in ANNI (47,027 lines of repository code; no PyTorch, cuDNN, or autograd), four runs on 2026-10-25 either diverged above the uniform-guess baseline of 9.72 nats or plateaued unexpectedly. A controlled 620-step probe comparing BF16 against FP32 on the same data and seed reached a training negative log-likelihood (NLL) of 6.060 in FP32 while exploding to 27.372 in BF16, leading to the initial diagnosis that BF16 matrix products were numerically unstable at the chosen learning rate. A subsequent end-to-end audit of the CUDA backend against the CPU reference implementation revealed that the precision comparison had measured the bugs rather than the number formats. The audit uncovered twelve defects in the GPU training path. Only one defect was specific to half-precision matrix products (a cuBLASLt algorithm-timing loop that executed accumulating weight-gradient GEMMs up to 16 times in place on the first step, inflating step-1 BF16 weight gradients up to 7.3x); the four most damaging defects acted in every precision, including FP32. Worst among them, cross-entropy target arrays were uploaded through a persistent device cache keyed on the host pointer address and refreshed only when a weight-epoch counter changed. Because the host target buffer was allocated with malloc per microbatch and recycled at the same heap address, every microbatch from the second optimizer step onward was scored and back-propagated against the targets of a single fixed microbatch—yet the reported training loss continued to fall for hundreds of steps as the model memorised token positions. We describe the residency and caching architecture where these twelve defects arose, give the exact mechanism, symptom, and fix for each defect alongside four additional instrumentation and data-path bugs found later, state the six framework-free invariants and regression tests that caught them, and show that after the fixes FP32 and BF16 agree to 0.0001 nats of validation loss over 150 optimizer steps and train 185M- and 555M-parameter models cleanly across billions of tokens.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon