Toward Testing AI: A Socio-Economic View (Technical Report)

<strong>Background: </strong>In industrial as well as societal contexts, there is a current trend to design arti!cial intelligence (AI) systems involving one or more (arti!cial) agents working alongside humans. Agents and humans interact to achieve a dedicated purpose. We assume that the interactions do not just emerge but are designed from a speci!c point of view. The analysis of the interactions, in particular to ensure systems are designed that are bene!cial for humans, is all but easy, and research to do the analysis in a methodological way is therefore urgently required. <strong>Objectives: </strong>This article presents an experimental methodology for analyzing the interaction of agents and humans. We assume that system analysis is conducted from two perspectives: From an economic and AI point of view, formal mechanism design is employed to model the interaction as a dynamic human-in-the-loop mechanism. From an analytical sociology viewing angle, social mechanism theory is employed to evaluate the mechanism’s causal, generative, normative and collective properties. <strong>Methods: </strong>In order to take the social and economic e"ects seriously in system analysis, it is argued that it is necessary to move beyond the well-known Turing test for agents, which focuses on a notion of intelligence as non-distinguishability of agents from humans. However, intelligence of agents does not necessarily imply bene!ciality or usefulness for humans. The core contribution of this article is the formulation of a Star-Durkheim-inspired test. <strong>Results: </strong>This test evaluates an AI system’s usefulness and ability to contribute to social mechanisms by focusing on collective cohesion (organic solidarity) and a possible dissolution of social structures (anomie) rather than focusing on isolated capabilities as the Turing test. <strong>Conclusions: </strong>Integrating economic mechanism design with analytical sociological paradigms provides a new perspective on AI systems and their applications. Furthermore, this theoretical underpinning aims to pave the way for social mechanism design theory and lays the groundwork for well-founded regulation of AI systems.