Risk for AI Agents

Abstract This paper investigates whether the concept of risk applies to AI agents. The EU AI Act, in force since 2024, grounds its regulatory architecture in a product-based model of risk: one that presupposes an intended purpose, (a bounded space of) reasonably foreseeable misuses, and a tractable distribution of potential harms. This paper argues that these three presuppositions fail structurally for sufficiently general agentic systems. Agents whose behavioural space is open-ended by design cannot be assigned a purpose specific enough to anchor risk identification; their misuse space is generatively open, including novel sequences discoverable by the agent itself; and the probability and severity of harm are subject to modification through the agent’s own learning and action on the environment. What fails, however, is the product-style operationalisation of risk (per system, ex ante , anchored in intended purpose), rather than risk as such. Given this diagnosis, the paper develops two paths for conceptualising agentic risk: a conservative path, which retains the standard definition of risk and reconstructs it formally for narrow agents (either as expected deviation from an ideal trajectory, or as the probability-weighted sum of constraint violations); and a governance path, which draws an analogy to the normative systems we use to manage biological agents, arguing that sufficiently general AI agents require something closer to law-following than to product-safety assessment. The two paths are compared along five dimensions and shown to be complementary: the first suited to operational risk monitoring within bounded deployments and the second to the governance architecture that determines which agents may legitimately be deployed at all.