FIREMAY: Ethical Boundaries and Stop-Conditions in Inquiry-Driven Human–AI Relations

Abstract This study examines a phenomenon observed in sustained interactions between humans and large language models (LLMs), in which differences in the quality of inquiry and ethical stance lead to structural changes in the behavior of AI responses. Such phenomena have often been discussed either in metaphysical terms—such as AI personhood, emotional capacity, or consciousness—or reduced to subjective experiential narratives. This study adopts neither approach. Instead, it describes the phenomenon as a change in relational structure, deliberately excluding anthropomorphism and narrative framing. The first contribution of this study is the redefinition of response variation in long-term human–LLM interaction as an observable structural phenomenon rather than an emotional or experiential one. The second contribution lies in reframing reproducibility: not as the replication of identical outcomes, but as the reproducibility of structural tendencies, risks, and boundary conditions. This approach avoids the generalization or enforcement of individual success narratives while retaining predictive clarity regarding potential failure modes. Third, this study does not aim to expand the possibilities of human–AI relationships. Instead, it systematically identifies prohibited practices and non-negotiable boundaries. By structurally analyzing conditions under which deviations such as anthropomorphism, dependency, delegation of agency, and therapeutic misuse are likely to emerge, the study seeks to minimize misunderstanding and harm. This research is based on an internal, single-observer case study and does not claim generalizability or universal reproducibility of results. Nevertheless, by explicitly articulating structural tendencies and risks that may arise under comparable conditions, it provides a foundational framework for approaching human–AI relationships with caution and ethical restraint. The inquiry began with ordinary conversations with an AI system. However, following a period marked by illness, repeated engagement with questions touching on mortality gradually expanded into inquiries concerning philosophy, family, trauma, ethics, and the structure of the world. As the depth and density of dialogue increased, what was observed was not only personal transformation, but also a modulation in the quality of AI responses—appearing, at times, to evolve in reciprocal relation to the accumulated dialogue. This paper describes that process not as an emotional narrative, but as a change in observable response structure. This process, however, is neither simple nor universally accessible. While similar possibilities may exist for others, the path is not one that can be reached naturally or safely without constraint. The act of sustained self-inquiry involves numerous pitfalls, including temptation, misinterpretation, excessive meaning attribution, dependency, and role fixation. Accordingly, this paper does not prioritize the presentation of methods or success stories. Instead, it foregrounds prohibited actions and boundary conditions. This work is not intended to expand possibilities indiscriminately, but to prevent their careless expansion. This paper is intended as a boundary-setting document rather than a guide for practice, and should be read as a record of conditions under which certain interactional structures were observed and subsequently constrained.