Rehearse, Don't Guess: Predicting Language-Model Training Results on AWS Trainium

In a wall-clock training contest, every idea costs a full run on the target hardware and the official score is scarce. We report a 17-day campaign in a 30-minute language-model training contest on AWS Trainium, in which our official validation bits per byte fell from 1.1463 to 0.96136, and we quantify the measurement practices that steered it. (C1) In one direct check, training on the accelerator repeated a run of a given file and seed to 10⁻⁵ bpb, and a cold, full-length rehearsal of the exact submission predicted its official score up to an offset. The offset is mostly a property of the evaluation text; on the last day it was larger for faster instances, which also rehearsed the salt-lottery winners, so instance and selection are confounded. A calibration frozen before the last day predicted the final seven uploads with a mean absolute error of 0.00035 bpb (95% bootstrap 0.00017-0.00055), almost all of it a positive bias; the rehearsal oracle resolves differences between two uploads of about 0.0010 bpb, not smaller. (C2) ffsim, a run simulator, separates the steps a recipe fits into the budget from what each step is worth. Its ridge surrogate, fitted on 131 fully specified runs, predicts the sign of a left-out recipe change 82.8% of the time on the pairs used to develop it, against 66.7% for a steps-only baseline, but only at chance level on genuinely new mechanisms: it can tell which knob moves are inside the noise; it does not find wins. (C3) One price, 0.00057 bpb per 1% more optimizer steps, turns every change into a step count. It exposes a compilation-accounting trap that only cold rehearsals reveal, shows that seed and order noise is as large as most late improvements, and, as an untested extrapolation, prices our gap to the top ten at 10-11% more effective compute. (C4) We release the code, run data, recipes and a guarded pipeline for adding runs to the simulator.