ANNI: A Transformer Language-Model Training System Written from Scratch in C and CUDA, and What Building It Taught
ANNI is a transformer language-model training system written from scratch in C and CUDA, with no machine-learning framework underneath: hand-derived forward and back-ward passes, a BPE tokenizer, Adam, checkpointing with mid-epoch resume, and a CUDA backend using cuBLAS and cuBLASLt (FP32, BF16, FP8) with hand-written kernels. This report describes the system and its runs on one 16 GB consumer GPU. A 184.7M-parameter model was trained for one epoch on 3.81 billion tokens to a validation loss of 3.42 nats per token (41 h 50 min of wall clock, 29 process starts, 28 resumes). A 555.8M-parameter model is in progress: at 66.4% of its epoch, with validation loss 3.22 and HellaSwag acc_norm 0.369. An audit of the GPU path, prompted by diverging runs, found twelve defects, mostly stale or mis-keyed device state, one of which let the loss fall while the model trained on fixed targets. After the fixes, FP32, BF16 and FP8 agree within 0.01 nats of validation loss over 150 steps. We state plainly what is missing: there is no comparison with any otherframework, every result is a single run with one seed, the 55M run is unfinished, and the chat models are fluent and often wrong.
Authors
- Vincent Abel
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24226037
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00