The AGI power dilemma and the illusion of safety with humans in the loop

Avoiding loss-of-control events will be key to industry and society as AI technology is employed in various critical processes. In this article, we assess a hypothetical instability situation of an AGI − a singularity event − from a safety science standpoint. We present one way to describe the process of how a real time Artificial General Intelligence (AGI) in its interaction with the environment may set it in a runaway condition. Our goal is clarifying general and structural control issues of strong real time AI for design, safeguarding and legislation. In cybernetics, instability and hence, the risk of loss-of-control, is a core trait of all highly adaptive systems. Such adaptive behavior can manifest as radical reframing – a pivotal process of a system that enables it to solve very hard challenges, and which requires it to redefine its core concepts, goals, values, plans and procedures. What emerges is the AGI power dilemma ; a double-bind between performance and controllability of a real time system. We discuss the consequences for human control of AGI and prospects for safeguarding when optimizing for power and speed. Moreover, we conclude that the human in the loop approach gives an illusion of safety when the AI is optimized for power.