From human-system interaction to human-system co-action and back: ethical assessment of generative AI and mutual theory of mind

Abstract Human-machine ethics has emerged as a rapidly growing research field in recent years. However, it seems that Generative Artificial Intelligence (AI) leads to a paradigm shift from human-machine interaction to co-action. The ethical assessment of such relationships is still in the making and needs further scrutiny. First, studies about the influence of technology in human-system interactions and manipulation are reviewed. Second, the “mutual theory of mind” approach is critically examined to identify its shortcomings. Third, creating user models is reconstruced to demonstrate the strategies of systems. Finally, use cases are discussed and assessed to outline ethical implications.