Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32

Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) relative to the final checkpoint, and relative to the best single checkpoint. Models are evaluated in fp32. Forty validation blocks select, and 100 disjoint test blocks report. Averages with k=2 to 5 lower the test NLL by 0.041 to 0.064 nats, with paired intervals below zero. The average at k=6 does not (-0.0019 nats). Each average also beats its own best member. However, the single 100k checkpoint already has NLL 3.5518, 0.0616 nats below the final one, and validation picks it. The best average (k=5) differs from it by -0.0023 nats [-0.0083,+0.0034]: no difference larger than 0.008 nats detected, without an equivalence test. Pythia was trained in fp16, and an earlier bfloat16 evaluation produced a spurious 0.45-nat gain. The evidence comes from one 160M-parameter model and one corpus.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35014003
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32

Dheiver Francisco Santos
Figshare
Topic Modeling
preprint

Averaging the Last Checkpoints of Pythia-160M: No Difference Larger Than 0.008 Nats From the Best Single Checkpoint, WikiText-2 NLL in fp32

Dheiver Francisco Santos
preprint en

Abstract

Public intermediate checkpoints let us average a language model's own training trajectory at no training cost. We ask whether averaging the last k of six Pythia-160M checkpoints (steps 143k to 100k) lowers WikiText-2 next-token negative log-likelihood (NLL) relative to the final checkpoint, and relative to the best single checkpoint. Models are evaluated in fp32. Forty validation blocks select, and 100 disjoint test blocks report. Averages with k=2 to 5 lower the test NLL by 0.041 to 0.064 nats, with paired intervals below zero. The average at k=6 does not (-0.0019 nats). Each average also beats its own best member. However, the single 100k checkpoint already has NLL 3.5518, 0.0616 nats below the final one, and validation picks it. The best average (k=5) differs from it by -0.0023 nats [-0.0083,+0.0034]: no difference larger than 0.008 nats detected, without an equivalence test. Pythia was trained in fp16, and an earlier bfloat16 evaluation produced a spurious 0.45-nat gain. The evidence comes from one 160M-parameter model and one corpus.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Figshare
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon