Why Does AI Companionship Go Wrong?

AI companions, powered by advanced language models, offer personalised interactions and emotional support, but their increasing prevalence raises significant ethical concerns. This paper examines the complex interplay of factors contributing to the potential negative impacts of AI companions in a case study. This author further argues that the root of the negative impacts comes from insufficient user screening that may expose vulnerable individuals to unsuitable AI interactions, regulatory frameworks struggling to keep pace with rapid technological advancements, and a lack of clear distinction between inherent AI limitations and temporary developmental artifacts. This paper aims to provide insights for responsible AI development, and calls for robust user screening protocols, adaptive regulatory frameworks and more informed research mindsets.