ANNI: A Transformer Language-Model Training System Written from Scratch in C and CUDA, and What Building It Taught

ANNI is a transformer language-model training system written from scratch in C and CUDA, with no machine-learning framework underneath: hand-derived forward and back-ward passes, a BPE tokenizer, Adam, checkpointing with mid-epoch resume, and a CUDA backend using cuBLAS and cuBLASLt (FP32, BF16, FP8) with hand-written kernels. This report describes the system and its runs on one 16 GB consumer GPU. A 184.7M-parameter model was trained for one epoch on 3.81 billion tokens to a validation loss of 3.42 nats per token (41 h 50 min of wall clock, 29 process starts, 28 resumes). A 555.8M-parameter model is in progress: at 66.4% of its epoch, with validation loss 3.22 and HellaSwag acc_norm 0.369. An audit of the GPU path, prompted by diverging runs, found twelve defects, mostly stale or mis-keyed device state, one of which let the loss fall while the model trained on fixed targets. After the fixes, FP32, BF16 and FP8 agree within 0.01 nats of validation loss over 150 steps. We state plainly what is missing: there is no comparison with any otherframework, every result is a single run with one seed, the 55M run is unfinished, and the chat models are fluent and often wrong.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24226037
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

ANNI: A Transformer Language-Model Training System Written from Scratch in C and CUDA, and What Building It Taught

Vincent Abel
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

ANNI: A Transformer Language-Model Training System Written from Scratch in C and CUDA, and What Building It Taught

Vincent Abel
article en

Abstract

ANNI is a transformer language-model training system written from scratch in C and CUDA, with no machine-learning framework underneath: hand-derived forward and back-ward passes, a BPE tokenizer, Adam, checkpointing with mid-epoch resume, and a CUDA backend using cuBLAS and cuBLASLt (FP32, BF16, FP8) with hand-written kernels. This report describes the system and its runs on one 16 GB consumer GPU. A 184.7M-parameter model was trained for one epoch on 3.81 billion tokens to a validation loss of 3.42 nats per token (41 h 50 min of wall clock, 29 process starts, 28 resumes). A 555.8M-parameter model is in progress: at 66.4% of its epoch, with validation loss 3.22 and HellaSwag acc_norm 0.369. An audit of the GPU path, prompted by diverging runs, found twelve defects, mostly stale or mis-keyed device state, one of which let the loss fall while the model trained on fixed targets. After the fixes, FP32, BF16 and FP8 agree within 0.01 nats of validation loss over 150 steps. We state plainly what is missing: there is no comparison with any otherframework, every result is a single run with one seed, the 55M run is unfinished, and the chat models are fluent and often wrong.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon