Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32
Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) relative to the final checkpoint, and relative to the best single checkpoint. Models are evaluated in fp32. Forty validation blocks select, and 100 disjoint test blocks report. Averages with k=2 to 5 lower the test NLL by 0.041 to 0.064 nats, with paired intervals below zero. The average at k=6 does not (-0.0019 nats). Each average also beats its own best member. However, the single 100k checkpoint already has NLL 3.5518, 0.0616 nats below the final one, and validation picks it. The best average (k=5) differs from it by -0.0023 nats [-0.0083,+0.0034]: no difference larger than 0.008 nats detected, without an equivalence test. Pythia was trained in fp16, and an earlier bfloat16 evaluation produced a spurious 0.45-nat gain. The evidence comes from one 160M-parameter model and one corpus.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.
Authors
- Dheiver Francisco Santos (ORCID: https://orcid.org/0000-0002-8599-9436)
Publication Details
- Journal
- Figshare
- Published
- 2026-11-15
- DOI
- https://doi.org/10.6084/m9.figshare.35014003
- Primary Topic
- Topic Modeling
- Type
- preprint