BEFORE AUTONOMY: Earning the Right to Act

If you have only five minutes, here is the whole book. This book argues for a simple engineering principle: a system should not receive more freedom to act than we have the independent ability to verify, constrain, and stop. Five ideas carry the argument. 1. Safety Must Be Designed Before Capability Growth Boundaries, stop mechanisms, verification criteria, and rules for expanding authority must exist before the system receives the corresponding freedom. “We’ll patch it later” does not work when the consequences may be irreversible. 2. A Claim Is Not Evidence Neither a developer’s statement (“we checked everything”) nor the system’s own statement (“I am safe”) constitutes engineering evidence. Trust must rest on checks that can be independently repeated, challenged, and traced to evidence. 3. Freedom Grows in Steps, Alongside Verifiability Autonomy is not a switch. It is a set of distinct authorities that can be granted, limited, or withdrawn. Each increase in authority requires its own evidentiary basis. As independent action becomes more consequential, verification must grow with it. If verification falls, granted freedom must fall with it. 4. No Self-Authorization—and No Safety Only from the Outside The system must not decide for itself when the evidence is sufficient to receive more authority. But safety cannot consist only of an external fence around an otherwise unrestricted capability. Control must also be built into the architecture and preserved as capabilities change. The goal is not to keep a system below some fixed level of intelligence. It is to ensure that control properties do not disappear as capability increases. 5. Unknowns Must Remain Visible The absence of an observed failure does not prove the absence of failure. An unassessed item is not a pass. Every safety claim has boundaries: configuration, conditions, scope, evidence, and validity period. What remains unknown must remain visible—and must not silently turn into authorization. In short: capability is not authority, authority is not operation, and a successful test is not a permanent guarantee. Don't take my word for it. Check it.