A Lightweight Selective-Routing Study for CPU-Only Inference

This paper studies a lightweight selective-routing framework designed for CPU-only inference on resource-constrained devices. Rather than introducing a new model family or complex predictive architecture, the system implements a model-agnostic orchestration layer built around an always-on guard model, an expert stage, and a cost-aware trigger. Easy inputs are handled cheaply by the guard, while more complex samples selectively trigger a short-lived burst of stronger computation before returning to the low-cost path. The framework evaluates three distinct routing signals: raw margin (uncertainty near the decision boundary), wobble (prediction instability under small input perturbations), and a combined signal fusing ambiguity and fragility. Through a sequence of experimental progressions across standard benchmarks (including Digits, Wine, and Breast Cancer datasets), we show that routing performance is highly regime-dependent. In balanced settings, raw margin serves as the strongest general-purpose trigger. However, under extreme class imbalance—evaluated on a real-world credit card fraud benchmark—the most robust, deployable policy is highly conservative: operating under a tight 0.1% trigger-budget cap, the wobble signal safely preserves guard-level performance (mean balanced accuracy of 0.9288 and mean positive recall of 0.8606) while maintaining an ultra-low mean trigger rate of 0.0006. Ultimately, this work characterizes the exact boundary conditions under which a tiny control policy can optimize the accuracy-efficiency frontier relative to strong, always-on baselines. To ensure complete reproducibility and maintain open science standards, links to the originating large language model (LLM) conversation logs and the experimental Google Colab execution notebooks are explicitly provided within the manuscript.