Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning

Sparse expert supervision can improve reinforcement learning, but deciding whichstates are worth annotating remains difficult. Uncertainty identifies where a learneris unsure, but does not directly identify where supervision will most stronglychange that learner. We introduce ALBERTA, a response-aware acquisition frame-work that measures local policy and value changes produced by controlled expertinterventions, predicts those responses from pre-annotation signals, and selectscandidates in the resulting two-dimensional response space. A calibration set of200 expert-labeled probes is used to learn response from policy entropy, historicalcritic disagreement, and their interaction; Pareto selection then allocates a separate121-state downstream budget. The response-aware branch therefore uses 321 ex-pert labels in total, compared with 121 for the Random baseline. In a controlledsingle-seed Atomic Chess experiment from a common RL10 branch point, thePareto branch achieves a higher mean game score than PPO-only at all four testedevaluation temperatures, with smaller margins at higher temperatures, and showsits largest margins over Random at lower temperatures. These results provideevidence that learner response can be a useful acquisition target while exposingsubstantial differences between policy and value responsiveness.