SAFE-SCALE: Unified Sub-Quadratic Architectures, Adaptive Test-Time Compute, and Mechanistic Steering for Long-Horizon Autonomous Agents
Long-horizon autonomous agents must keep execution histories that run to millions of tokens, spend test-time compute on deliberate multi-step search, and resist adversarial content arriving through tool outputs. These requirements conflict. Softmax attention incurs compute quadratic in trajectory length and a key-value (KV) cache that grows linearly per layer; tree search multiplies that state by the branching factor; and external guardrail models add serial latency at every tool boundary. We call this tension the Agentic Trilemma of capability, cost, and safety. We present SAFE-SCALE, a backbone-to-inference framework that resolves it by co-design rather than composition. (i) A hybrid backbone interleaves Gated DeltaNet and Mamba-2 layers with block-sparse attention at a 7:1 ratio, distilled from a dense 70B teacher, reducing the per-branch fork state to a constant ≈0.29 GB of recurrent state plus a shared, sparsely read attention cache. (ii) A difficulty-aware router invokes Process Reward Model (PRM)-guided speculative tree search only on high-uncertainty steps, exploiting constant-size state snapshots for O(1) fork and rollback. (iii) Sparse Autoencoders trained jointly on residual streams and SSM states expose monosemantic features for prompt injection, goal hijacking, and unauthorized tool invocation; a fused linear probe clamps these features and prunes compromised branches before any tool call is emitted. On SWE-bench Verified, RULER at 1M tokens, and AgentDojo, SAFE-SCALE targets 61.8% resolution, 84.1 RULER-1M accuracy, and an attack success rate of 2.9% (versus 24.7% undefended), while reducing peak VRAM by 5.3× and end-to-end latency by 2.7× relative to a dense 70B Transformer running the same search budget.
Authors
- Prateek Dutta (ORCID: https://orcid.org/0000-0002-9363-8067)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24414228
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- preprint