Dropping Anchor or Chasing the Horizon? Theoretical and Practical Challenges for Personalized AI Advisors

Abstract Unlike generic AI advisors which aid in normative deliberation according to preloaded values and creeds (i.e., Singerian Utilitarianism, Calvinist Protestantism, or Stoicism), personalized AI advisors aim to aid in users’ decision-making by their own lights . In this paper, I argue personalized AI advisors face a challenge called the Anchoring Problem : the difficulty of adjudicating between competing temporal and psychological reference points for normative guidance—whether to “chase the horizon,” defined as dynamically calibrating to whichever aspirational self or set of beliefs, dispositions, or values, a user presently holds, or to “drop anchor,” defined as stubbornly preserving whichever values the user set out with. This problem is compounded by several observations about aspirational selfhood, namely that a person’s conception of their aspirational self may be underdetermined and the identity it purports to describe may be incoherent, romanticized, or contradictory. Even if this is not the case, the aspirational self is rarely diachronically stable such that who one wishes to be or how one wishes to improve remains static over time. In this paper, I argue personalized AI advisors promise a number of advantages over generic ones if the Anchoring Problem proves surmountable, and I argue it is within certain constraints. Namely, I argue for the necessity of a ‘co-reasoning,’ dialectical model of personalization. Co-reasoning should safeguard the user against value hijacking by encouraging their deliberative and decisional participation while also pushing them to confront unaccounted discrepancies between stated values, aspiring values, and actual behavior (both past and present).