Dual manipulation of individuals by non-invasive intelligent technologies and its risk governance: from the perspective of the Destiny Machine metaphor

Introduction The “Destiny Machine” metaphor reveals an ethical concern: big data and artificial intelligence, operating as a data-power social machine, manipulate individuals with pre-determinacy as the core logic. At cognitive-information and behavioral decision-option levels, this manipulation appears as cognitive manipulation by recommendation algorithms and decision-making manipulation by automated decision systems. To uncover its psychological mechanisms and ethical risks, this paper builds a framework integrating ethics and psychology. Methods The Destiny Machine is transformed into an analytical tool with three dimensions: pre-determinacy, irresistibility, and concealment. Drawing on dual-process theory, automation bias, moral disengagement, and psychological reactance, it analyzes how non-invasive intelligent technologies psychologically manipulate individuals and shape their ethical risks, adopting the human-centered approach as the guiding principle for governance. Results Cognitive manipulation by recommender systems, typified by TikTok, leaves users unaware that their cognitive preferences are being systematically reshaped, keeping them imperceptibly immersed. Algorithms achieve this by activating Type 1 processing while bypassing Type 2 scrutiny, confining information to relevance channels, and exploiting automation bias and complacency to lower cognitive vigilance and suppress psychological reactance. Indulging in information cocoons widens users’ cognitive distance from the lifeworld, dissolving their moral attributes as ethical beings. Decision-making manipulation by automated decision systems is most evident in autonomous vehicles’ trolley-problem dilemmas, where passengers’ lives are reduced to quantifiable weights. The system pre-sets decision rules and allows remote intervention at the design end; opaque decision-making subjects and logic, millisecond-level deadlines, and impossibility of exit jointly pre-lock passengers’ fate before the accident occurs. It likewise exploits cognitive processing traits and automation bias, while dispersed agency makes responsibility attribution—and thus psychological reactance—difficult. Pre-coding an individual’s fate at the behavioral decision-option level may deconstruct their meaning world. Discussion Under the Destiny Machine metaphor, such manipulation erodes not only individuals’ autonomy but also their responsibility as moral subjects. Guided by the human-centered approach, it proposes governance schemes with clear actors, sound measures, and reliable benchmarks: a cognitive immune system of transparency and empowerment for cognitive manipulation, and a human-machine collaborative scheme of accountability and remedy for decision-making manipulation, rebuilding passengers’ trust to help them escape.