Memory Is Not a Policy: A Pre-Registered Test of Whether a Small Model Trained on a User's Own Choices Beats a Frontier LLM Given the Same History

Portable, user-controlled context stores for AI assistants now exist as commercial products (e.g., Egoist Machines' AI Passport). A personalization system proposing "another memory layer" must therefore explain why a well-prompted frontier model reading that same store would not already suffice. This paper argues that the only defensible point of difference is a trainable decision policy: a small model fit to a specific user's revealed choices, rather than a retrieval system that hands text to a large model and hopes it infers preferences. We set this out as a pre-registered protocol. The experiment holds the information constant and varies only the mechanism: a small supervised model trained on a user's past pairwise choices is compared against a frontier LLM that receives the identical history in its context window, both predicting held-out choices from the same user. We specify the datasets, conditions, metrics, sample sizes, and decision rule in advance. If the trained policy does not beat the in-context LLM by a pre-set margin at realistic history lengths, the policy-based approach loses its technical justification. We also provide a threat analysis and a reporting template for the completed study. No results are claimed; the contribution is the design and the commitment to its outcome.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-29
DOI
https://doi.org/10.5281/zenodo.23770781
Primary Topic
Recommender Systems and Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Memory Is Not a Policy: A Pre-Registered Test of Whether a Small Model Trained on a User's Own Choices Beats a Frontier LLM Given the Same History

Atharv Sharma
1 links
Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
preprint

Memory Is Not a Policy: A Pre-Registered Test of Whether a Small Model Trained on a User's Own Choices Beats a Frontier LLM Given the Same History

Atharv Sharma
preprint en

Abstract

Portable, user-controlled context stores for AI assistants now exist as commercial products (e.g., Egoist Machines' AI Passport). A personalization system proposing "another memory layer" must therefore explain why a well-prompted frontier model reading that same store would not already suffice. This paper argues that the only defensible point of difference is a trainable decision policy: a small model fit to a specific user's revealed choices, rather than a retrieval system that hands text to a large model and hopes it infers preferences. We set this out as a pre-registered protocol. The experiment holds the information constant and varies only the mechanism: a small supervised model trained on a user's past pairwise choices is compared against a frontier LLM that receives the identical history in its context window, both predicting held-out choices from the same user. We specify the datasets, conditions, metrics, sample sizes, and decision rule in advance. If the trained policy does not beat the in-context LLM by a pre-set margin at realistic history lengths, the policy-based approach loses its technical justification. We also provide a threat analysis and a reporting template for the completed study. No results are claimed; the contribution is the design and the commitment to its outcome.

Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon