Beyond optimization: power, rationality and ethical risk assessment of AI

Abstract Emerging technologies pose various ethical challenges, including the need to assess their risks adequately. Still, the roles of different forms of rationality and power in assessing risks have not been fully scrutinized. To address this, we must differentiate forms of rationality to conceptualize power: instrumental, technical, and probabilistic. The latter has gained more power through the rise of artificial intelligence (AI), focusing mainly on optimization and accuracy. These forms of power and rationality lead to different conceptualizations of risk: real, hypothetical, and predictive risk. The ethical issues surrounding AI have not yet been fully scrutinized, since they are mainly approached through a lens of probabilistic rationality. I argue that optimizing algorithms to increase accuracy or engagement distracts from assessing whether AI-based systems are adequate for certain tasks. Furthermore, I discuss ways to govern AI-based systems through participatory processes and standardization that enable control and autonomous judgment in the deployment and use of such systems.