Probe-Then-Commit: Constrained Reinforcement Learning for Adaptive Precision and Early-Exit Inference on Resource-Constrained Edge Devices

Transformer encoders deployed on edge devices must answer each query within a latency deadline while the energy budget, temperature, and memory bandwidth of the device change over time. Hand-designed runtime managers switch among pre-compiled precision modes with feedback rules, but they react to the device rather than to the input and do not plan how a session budget is spent. We cast adaptive inference as a constrained Markov decision process and propose a probe-then-commit controller trained with proximal policy optimization and PID-Lagrangian multipliers. For every query the policy first selects the precision of a three-layer probe, reads the normalized entropy of the probe’s exit classifier, and then either answers immediately or commits a suffix precision and an exit threshold. A feasibility shield masks actions that would violate the response deadline or an energy reserve, and a behavior-cloning warm start from a myopic pacing heuristic stabilizes learning. The controller is trained in a trace-driven environment built from real per-utterance outputs of a RoBERTa-base intent classifier on SLURP and from a batch-1 latency table measured on a CPU core (8.9% MAPE against end-to-end timing). The measurements show that natively supported INT8 and BF16 kernels run 7–15× faster than software-unpacked 4-bit weights and that the layer-9 exit is more accurate than the full network. Across five held-out device scenarios the controller attains 87.00% mean accuracy with no budget depletion, compared with 86.75% for a tuned greedy pacing controller, 86.46% for a tuned re-implementation of a PID runtime manager, and 85.35% for the best static rule.