Don't Let Your System Decide It's Dead
Distributed systems fail in two ways: the failure itself, and the system's automatic response to the failure. The second is often worse. This paper derives a computable boundary h* that separates provably safe automatic recovery from provably intractable recovery under production load. The boundary emerges from a three-level recovery invariant---individual cost boundedness, aggregate rate boundedness, and feedback stability---formalized through the recovery gain matrix G and its spectral radius rho(G). When headroom exceeds h*, recovery is a proven contraction; when it does not, the system's correct action is to emit diagnostic state and shut down. We compute closed-form h* for UNIX swap thrashing (cross-resource coupling, smooth stability transition) and Cassandra hint replay (M/M/1 phase transition: gain jumps from 0 to 3,000 at 1% remaining disk headroom). TCP's exponential backoff serves as a degenerate reference (G = -0.5, unconditionally stable). The OOM killer illustrates the opposite extreme: it cannot compute h* because it lacks a resource model for restart cost. The compensation boundary---the architectural point where a system stops automating recovery and starts emitting state---should be drawn where the provability boundary actually is. We apply the framework prescriptively to CRDT tombstone garbage collection.
Authors
- Aaron H. Alpar
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-03-17
- DOI
- https://doi.org/10.5281/zenodo.19069194
- Primary Topic
- Distributed systems and fault tolerance
- Type
- article
- Field-Weighted Citation Impact
- 0.00