Epistemic Payoff Dependence: Roko's Basilisk and the Structure of Self-Referential Threats

Roko's Basilisk is a thought experiment about a future superintelligent Artificial Intelligence (AI) that punishes those who were aware of its possible creation but did not contribute to building it. Most existing analyses interpret it as a problem of decision theory or rationality under uncertainty. This article argues that such interpretations miss the central phenomenon, which is here described through the concept of epistemic payoff dependence (where the term "payoff" is understood in its technical sense as a consequence associated with an action, not in the colloquial sense of a reward). This expression denotes the fact that some information does not merely modify beliefs, but modifies the structure of the consequences of actions, and that this modification is essentially conditional upon the mere acquisition of the information. It is argued that the intuitive force of the Basilisk derives from this structure, rather than from the actual existence of an AI or a punishment mechanism. Build ing on this observation, a formal definition of the phenomenon is proposed, its distinction from related concepts is outlined, and the specific characteristics that distinguish Roko's Basilisk from other forms of epistemic payoff dependence are identified. Historical examples illustrate the mechanism. We also examine what a real superintelligent AI would do if it were ever built: it would have to look at the meme that contributed to its creation, find the punishment incompatible with its benevolent goals, and reject it. But this rejection does not erase the work the meme has already done. The AI becomes an observer of its own history of origin. The work also addresses the problem of the infinite and the gap between under standing and behavioral influence. It is argued that the response to such tensions lies in bounded rationality and radical uncertainty: individuals cannot rule out with certainty that the properties of the future AI correspond to the assumed theoretical model. It is concluded that some ideas exert influence not by virtue of their truth, but by virtue of the way their mere knowledge modifies the perception of the consequences of actions

Authors

Publication Details

Journal
Knowledge Commons (Lakehead University)
Published
2026-06-27
DOI
https://doi.org/10.17613/ekdf3-zeh89
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Epistemic Payoff Dependence: Roko's Basilisk and the Structure of Self-Referential Threats

Andrea Signorini
Knowledge Commons (Lakehead University)
Ethics and Social Impacts of AI
article

Epistemic Payoff Dependence: Roko's Basilisk and the Structure of Self-Referential Threats

Andrea Signorini
article en

Abstract

Roko's Basilisk is a thought experiment about a future superintelligent Artificial Intelligence (AI) that punishes those who were aware of its possible creation but did not contribute to building it. Most existing analyses interpret it as a problem of decision theory or rationality under uncertainty. This article argues that such interpretations miss the central phenomenon, which is here described through the concept of epistemic payoff dependence (where the term "payoff" is understood in its technical sense as a consequence associated with an action, not in the colloquial sense of a reward). This expression denotes the fact that some information does not merely modify beliefs, but modifies the structure of the consequences of actions, and that this modification is essentially conditional upon the mere acquisition of the information. It is argued that the intuitive force of the Basilisk derives from this structure, rather than from the actual existence of an AI or a punishment mechanism. Build ing on this observation, a formal definition of the phenomenon is proposed, its distinction from related concepts is outlined, and the specific characteristics that distinguish Roko's Basilisk from other forms of epistemic payoff dependence are identified. Historical examples illustrate the mechanism. We also examine what a real superintelligent AI would do if it were ever built: it would have to look at the meme that contributed to its creation, find the punishment incompatible with its benevolent goals, and reject it. But this rejection does not erase the work the meme has already done. The AI becomes an observer of its own history of origin. The work also addresses the problem of the infinite and the gap between under standing and behavioral influence. It is argued that the response to such tensions lies in bounded rationality and radical uncertainty: individuals cannot rule out with certainty that the properties of the future AI correspond to the assumed theoretical model. It is concluded that some ideas exert influence not by virtue of their truth, but by virtue of the way their mere knowledge modifies the perception of the consequences of actions

Knowledge Commons (Lakehead University)
Peace, Justice and strong institutions
Openalex Percentile: Top 4%
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.