How to Protect Learner Autonomy in the Age of Generative AI: EMM

Abstract Generative AI is increasingly used to interpret feedback and prepare consequential educational conversations, yet manipulation benchmarks largely evaluate models as external observers. The gap is particularly salient in student supervision, where an assistant drafting a student’s reply must recognize unreasonable pressure without treating legitimate academic demands as abuse. To fill this gap, we introduce EMM (E*** Mental Manipulation), an education-centered benchmark that pairs first-person student reply generation with 14-way observer-style classification on the same advisor utterances. Its educational set contains 1,014 Chinese-language cases: 676 manipulative utterances across 13 categories and 338 matched healthy, high-pressure controls. We extend the protocol to workplace and parent-child relationships, yielding 3,042 evaluation cases, and provide a disjoint 11,521-example three-domain SFT corpus. Across 15 model configurations, we find a recognition–response gap: an accurate manipulation label does not always lead to an aware first-person reply, while high awareness can coincide with over-defensiveness on healthy cases. Three-domain SFT improves manipulation awareness for three matched models but reduces their healthy-case calibration. A 2×2 ablation shows that third-person generation does not consistently outperform first-person generation; a minimal-prompt ablation shows that tuned models retain more awareness without explicit boundary guidance but remain prompt-sensitive. EMM provides an audit protocol for evaluating recognition and proportionate response together in learner-facing AI. Whether better benchmark performance protects student autonomy in practice requires further study.