Risk-Aware Self-Play Reinforcement Learning for Secure and Capable Code-Generating Agents

Abstract We study risk-aware self-play in finite-action code repair, separating functional success from test-integrity and forbidden-call violations. From 591 MBPP/HumanEval programs, 395 mutation tasks yield 4,527 candidates. On 133 external HumanEval tasks, the full policy reduces expected violations from 68.55% to 0.64% versus public-test-only RL, but fixed multiobjective rewards achieve slightly higher joint success, 72.78% versus 72.37%. Additional experiments execute 5,296 alias/data-flow probes and matched controls, investigate selected-action sparse verification, and test 30 functions from Python, JavaScript and TypeScript repositories. Detector blind spots, identical safe/unsafe feature representations and weak repository transfer limit the safety interpretation. Zero greedy violations on 133 tasks yields only an illustrative 2.23% upper risk bound under iid assumptions. These results support verifier-aware rewards without establishing a consistent self-play advantage or a formal deployment guarantee.