AI survival stories, types of risk, and the precautionary principle

Cappelen, Goldstein, and Hawthorne’s article offers a refreshing new perspective on questions of AI safety. Debates over AI and existential risk standardly start from a background assumption of humanity’s continued survival, and then reason about the probability of catastrophic outcomes. This paper, by contrast, flips that mode of reasoning on its head: a decision-maker should start from the assumption that powerful AI systems will destroy humanity, and then reason about the different outcomes in which humanity is saved from such an existential threat. Doing so enables us to better partition the space of possible events in which saving occurs, and so to come to more justified probabilities of humanity being saved. The article also illustrates the value of this approach by defending particular claims about promising paths to avert existential risk. To do so, it puts forth a model that different parties to the debate can use to calculate the probability that humanity is destroyed. The authors also defend a number of claims about the most likely mechanisms for realizing each of the four survival stories.