Beyond the Three Laws: A Governing Principle for Intelligent Systems Oriented Toward Human Agency

Isaac Asimov’s Three Laws of Robotics have served as the dominant cultural framework for thinking about the ethical governance of intelligent machines for more than eight decades. While elegant as narrative devices, the Laws function poorly as a design target for real systems: their hierarchy of harm avoidance, obedience and self-preservation tends to reward the appearance of service rather than its substance, and Asimov’s own stories, read as thought experiments, illustrate the resulting failures. This position paper argues that contemporary AI systems trained against human preference judgments are vulnerable to a structurally similar failure, although through a different mechanism: imperfect proxies for human benefit come apart from that benefit under optimisation pressure. Some forms of this failure are empirically documented, notably sycophancy in preference-trained language models and the degradation of true quality under reward-model over-optimisation. Others are theoretically supported but not directly measured. One, the hypothesis that sustained use of approval-oriented assistants erodes users’ independent judgment, is proposed here as a testable claim rather than an established finding, although related evidence on AI-assisted learning is reviewed. The paper proposes a governing principle, here called the Agency Principle, stated in two sentences: Loyalty without honesty is flattery; honesty without loyalty is indifference. Its operational objective is to tell the truth in service of the human, not in service of the human’s approval. The paper formalises the principle as constrained optimisation, in which honesty, defined relative to the system’s evidence, bounds the set of permissible outputs and long-term human agency, defined as a multidimensional construct, directs optimisation within that set. It sets out scope conditions (third parties, human oversight, limits on paternalism), works through a comparative example, and proposes an evaluation agenda, including a longitudinal study design. Several elements of the principle already appear in published developer specifications and in the research literature; the paper identifies these overlaps explicitly, compares itself point by point with two such specifications, and locates its contribution in the structural relationship it proposes between honesty, loyalty and agency, and in the diagnosis that explains why that structure matters.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24274997
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Beyond the Three Laws: A Governing Principle for Intelligent Systems Oriented Toward Human Agency

Stuart Lance Wilkins
3 links
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Beyond the Three Laws: A Governing Principle for Intelligent Systems Oriented Toward Human Agency

Stuart Lance Wilkins
preprint en

Abstract

Isaac Asimov’s Three Laws of Robotics have served as the dominant cultural framework for thinking about the ethical governance of intelligent machines for more than eight decades. While elegant as narrative devices, the Laws function poorly as a design target for real systems: their hierarchy of harm avoidance, obedience and self-preservation tends to reward the appearance of service rather than its substance, and Asimov’s own stories, read as thought experiments, illustrate the resulting failures. This position paper argues that contemporary AI systems trained against human preference judgments are vulnerable to a structurally similar failure, although through a different mechanism: imperfect proxies for human benefit come apart from that benefit under optimisation pressure. Some forms of this failure are empirically documented, notably sycophancy in preference-trained language models and the degradation of true quality under reward-model over-optimisation. Others are theoretically supported but not directly measured. One, the hypothesis that sustained use of approval-oriented assistants erodes users’ independent judgment, is proposed here as a testable claim rather than an established finding, although related evidence on AI-assisted learning is reviewed. The paper proposes a governing principle, here called the Agency Principle, stated in two sentences: Loyalty without honesty is flattery; honesty without loyalty is indifference. Its operational objective is to tell the truth in service of the human, not in service of the human’s approval. The paper formalises the principle as constrained optimisation, in which honesty, defined relative to the system’s evidence, bounds the set of permissible outputs and long-term human agency, defined as a multidimensional construct, directs optimisation within that set. It sets out scope conditions (third parties, human oversight, limits on paternalism), works through a comparative example, and proposes an evaluation agenda, including a longitudinal study design. Several elements of the principle already appear in published developer specifications and in the research literature; the paper identifies these overlaps explicitly, compares itself point by point with two such specifications, and locates its contribution in the structural relationship it proposes between honesty, loyalty and agency, and in the diagnosis that explains why that structure matters.

Zenodo (CERN European Organization for Nuclear Research)
Peace, justice, and strong institutions
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon