Beyond Optimizer Bytes: A Unified Formalism for Low-Bit Training

Optimizer-state memory is a useful measure of low-bit training methods, but it does not tell us whether two trainers retain the same information or follow the same training process. We develop a framework for comparing trainers that makes these questions explicit. We model trainers as finite, typed stochastic transition systems and specify the behavior or quantity that each comparison must preserve. We state sufficient conditions for preserving the distribution of observable training traces, and separately for preserving specified quantities along individual traces. We apply the framework to ternary training mechanisms and a continuous-master straight-through-estimator reference. The examples show that a compact representation can preserve selected optimizer histories without preserving rounded parameter updates. They also distinguish reductions in stored state from changes in training quality and execution speed. Illustrative experiments support these distinctions but do not establish a general ranking of methods. The framework makes explicit what a claim of equivalence or compression requires, which assumptions support it, and what remains unproven.