Will power-seeking AGIs harm human society?

Abstract Many have argued, based on the Instrumental Convergence Thesis, that Artificial General Intelligences (AGIs) will exhibit power-seeking behavior. Such behavior, they warn, could harm human society and pose existential threats—namely, the risk of human extinction or the permanent collapse of civilization. These arguments often rely on an implicit and underexamined assumption: that AGIs will develop world models—internal representations of world dynamics—that resemble those of humans. We challenge this assumption. We argue that once the anthropomorphic assumption—that AGIs’ world models will mirror our own—is rejected, it becomes unclear whether AGIs would pursue the types of power commonly emphasized in the literature, or any familiar types of power at all. This analysis casts doubt on the strength of existing arguments linking the Instrumental Convergence Thesis to existential threats. Moreover, it reveals a deeper layer of uncertainty. AGIs with non-human world models may identify novel or unanticipated types of power that fall outside existing taxonomies, thereby posing underappreciated risks. We further argue that world model alignment—an issue largely overlooked in comparison with value alignment—should be recognized as a core dimension of AI alignment. We conclude by outlining several open questions to inform and guide future research.