Augur: Measuring When Predicting Data Makes It Faster to Move

Can a model that predicts data make it faster to store and transfer? We study two forms of prediction under one framework: predicting which item is requested next, so it can be prefetched into a bounded cache, and predicting the content of the data itself, so that sender and receiver share a model and only the surprises are sent. We built augur, an open-source, user-space testbed with a byte-bounded cache, sequential, stride and Markov predictors, a confidence-gated prefetch scheduler, a trace-driven storage simulator, and an exact, integer-only context-mixing codec (augur-cm). On ten real files, augur-cm was smallest on six of the seven compressible files, 6.8-31.5% smaller than the best of gzip, bzip2, xz, zstd and brotli, and 5.2% larger than xz on one executable. Every output decoded byte for byte, and incompressible input cost one extra byte. The price is speed: augur-cm decodes at about 0.5 MB/s, over a thousand times slower than zstd. On a link, it is the fastest option only below roughly 0.04-0.11 Mbit/s, and it would need 16-53x faster decoding to win at 2 Mbit/s. In simulated storage workloads, history-based prefetching helped only where access history repeated (1.08-1.10x for the default predictor), while trusting single observations slowed workloads to as little as 0.36x. Replaying the workloads with real reads on a hard disk with a cold page cache, a learned history-based predictor never beat a plain LRU cache by more than 1.03x (fixed-order epochs) and ran at 0.94x on recurring routines, where it read 28% more data; a clairvoyant predictor reached 1.21-1.31x on the non-sequential workloads, which bounds what access prediction could gain on that drive. Prediction therefore saves bytes reliably but saves time only when its compute and misprediction costs are small compared with the time the bytes would take to move. Code, corpus builder and all measurements: https://github.com/CygnusXl/Augur (version 0.3.0). Experiment design, code and drafting were carried out with substantial assistance from Claude, an AI assistant from Anthropic. The author is responsible for the content.