FisherShield: Sequential D-Optimal Anti-Rewards for Deployment-Time Reward Privacy in Reinforcement Learning

Reinforcement learning is increasingly deployed in autonomous systems whose reward functions encode operational priorities and decision-making objectives. However, observable behavior can enable external observers to infer sensitive reward parameters through inverse reinforcement learning. Existing deployment-time defenses mainly rely on action randomization or deception, without explicitly accounting for how previously released behavior affects the additional reward-sensitive information contributed by subsequent actions. To address this gap, we propose FisherShield, a deployment-time mechanism that balances task value against marginal reward-information exposure. Under a specified reference observer model, FisherShield computes the sensitivity of action log-likelihoods to the private reward and accumulates the sensitivity directions of released actions in an empirical information matrix. For each candidate action, the increase in the matrix’s log-determinant measures its marginal reference-score exposure and is converted into a weighted penalty subtracted from the task value. Updating the matrix after each release makes the penalty depend on the observation history, enabling sequential exposure control without retraining the task policy. Experiments show that FisherShield reduces cumulative reference-score log-det exposure by 26.4% in FourRooms and 34.3% in LavaGrid relative to Expert while preserving near-expert utility. The results are limited to the specified reference-model Bayesian observer and do not establish robustness to model misspecification or a defense-aware attacker.