Don't Let Your System Decide It's Dead

Distributed systems fail in two ways: the failure itself, and the system's automatic response to the failure. The second is often worse. This paper derives a computable boundary h* that separates provably safe automatic recovery from provably intractable recovery under production load. The boundary emerges from a three-level recovery invariant---individual cost boundedness, aggregate rate boundedness, and feedback stability---formalized through the recovery gain matrix G and its spectral radius rho(G). When headroom exceeds h*, recovery is a proven contraction; when it does not, the system's correct action is to emit diagnostic state and shut down. We compute closed-form h* for UNIX swap thrashing (cross-resource coupling, smooth stability transition) and Cassandra hint replay (M/M/1 phase transition: gain jumps from 0 to 3,000 at 1% remaining disk headroom). TCP's exponential backoff serves as a degenerate reference (G = -0.5, unconditionally stable). The OOM killer illustrates the opposite extreme: it cannot compute h* because it lacks a resource model for restart cost. The compensation boundary---the architectural point where a system stops automating recovery and starts emitting state---should be drawn where the provability boundary actually is. We apply the framework prescriptively to CRDT tombstone garbage collection.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-03-17
DOI
https://doi.org/10.5281/zenodo.19069194
Primary Topic
Distributed systems and fault tolerance
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Don't Let Your System Decide It's Dead

Aaron H. Alpar
Zenodo (CERN European Organization for Nuclear Research)
Distributed systems and fault tolerance
article

Don't Let Your System Decide It's Dead

Aaron H. Alpar
article en

Abstract

Distributed systems fail in two ways: the failure itself, and the system's automatic response to the failure. The second is often worse. This paper derives a computable boundary h* that separates provably safe automatic recovery from provably intractable recovery under production load. The boundary emerges from a three-level recovery invariant---individual cost boundedness, aggregate rate boundedness, and feedback stability---formalized through the recovery gain matrix G and its spectral radius rho(G). When headroom exceeds h*, recovery is a proven contraction; when it does not, the system's correct action is to emit diagnostic state and shut down. We compute closed-form h* for UNIX swap thrashing (cross-resource coupling, smooth stability transition) and Cassandra hint replay (M/M/1 phase transition: gain jumps from 0 to 3,000 at 1% remaining disk headroom). TCP's exponential backoff serves as a degenerate reference (G = -0.5, unconditionally stable). The OOM killer illustrates the opposite extreme: it cannot compute h* because it lacks a resource model for restart cost. The compensation boundary---the architectural point where a system stops automating recovery and starts emitting state---should be drawn where the provability boundary actually is. We apply the framework prescriptively to CRDT tombstone garbage collection.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 22%
Distributed systems and fault tolerance
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.