Memory Is Not a Policy: A Pre-Registered Test of Whether a Small Model Trained on a User's Own Choices Beats a Frontier LLM Given the Same History
Portable, user-controlled context stores for AI assistants now exist as commercial products (e.g., Egoist Machines' AI Passport). A personalization system proposing "another memory layer" must therefore explain why a well-prompted frontier model reading that same store would not already suffice. This paper argues that the only defensible point of difference is a trainable decision policy: a small model fit to a specific user's revealed choices, rather than a retrieval system that hands text to a large model and hopes it infers preferences. We set this out as a pre-registered protocol. The experiment holds the information constant and varies only the mechanism: a small supervised model trained on a user's past pairwise choices is compared against a frontier LLM that receives the identical history in its context window, both predicting held-out choices from the same user. We specify the datasets, conditions, metrics, sample sizes, and decision rule in advance. If the trained policy does not beat the in-context LLM by a pre-set margin at realistic history lengths, the policy-based approach loses its technical justification. We also provide a threat analysis and a reporting template for the completed study. No results are claimed; the contribution is the design and the commitment to its outcome.
Authors
- Atharv Sharma
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-29
- DOI
- https://doi.org/10.5281/zenodo.23770781
- Primary Topic
- Recommender Systems and Techniques
- Type
- preprint