A Modular Reinforcement Learning Framework for Iterative FPS Agent Development

Deep reinforcement learning (DRL) has been widely adopted to solve decision-making problems in complex environments, demonstrating high performance across various domains. However, DRL-based FPS agents are typically trained with a traditional, monolithic policy that integrates heterogeneous functionalities into a single network. This design hinders policy interpretability and severely limits structural flexibility, since even minor design changes in the action space often necessitate complete retraining of the entire network. These constraints are particularly problematic in game development, where behavioral characteristics are distinct and design updates are frequent. To address these issues, this study proposes a Modular Reinforcement Learning (MRL) framework. Unlike monolithic approaches, this framework decomposes complex agent behaviors into semantically distinct action modules, such as movement and attack, which are optimized in parallel with specialized reward structures. Each module learns a policy specialized for its own behavioral characteristics, and the final agent behavior is obtained by combining the outputs of these modules. This modular design enhances structural flexibility by allowing selective modification and retraining of specific functions, thereby reducing the inefficiency associated with retraining a monolithic policy. Experimental results on the 1-vs-1 training map show that the proposed modular agent achieves a maximum win rate of 83.4% against a traditional monolithic policy agent, demonstrating superior in-game performance. In addition, the retraining time required for modifying specific behaviors is reduced by up to 30%, confirming improved efficiency for development environments that require iterative behavioral updates.