The impact of reward scalarization and weight scheduling on optimization dynamics in multi-objective molecular design

Abstract Inverse molecular design imposes multiple property constraints, requiring them to be combined into a scalar reward for reinforcement-learning (RL) fine-tuning; a step that can be highly sensitive to the reward formulation. Here, we systematically analyze how scalarization choices and stabilization mechanisms shape optimization dynamics in RL fine-tuning for multi-parameter optimization (MPO). We evaluate three scalarization schemes (arithmetic mean, geometric mean, and Chebyshev) under multiple target regimes that vary in constraint tightness and prior support. While broad targets yield stable behavior across scalarizations, narrow and weakly supported targets expose sharp failure modes. We further show that these failure modes can be prevented through the usage of target-aware weight scheduling. Collectively, our results highlight critical interactions between scalarization choice and weight dynamics, and provide mechanistic insight and actionable guidance for stabilizing RL fine-tuning in constrained molecular MPO. Scientific contribution This work examines different reward formulations in multi-objective reinforcement learning for molecular design, going beyond the designs typically considered in prior studies. We further show how these formulations influence convergence behavior and collapse modes, and how their effects depend on the size and constraint structure of the multi-dimensional optimization space.