Using Dynamic Programming and Reinforcement Learning for Exploring Tradespaces in Changeability Assessment

Abstract The construction machinery industry faces many uncertainties stemming from environmental, operational, and market-related factors. To mitigate future risks, development teams often favor more broadly applicable solutions compared to localized performance gains in specific scenarios. This situation highlights the necessity of incorporating changeability in these solutions for developing value-robust systems that can manage future uncertainty. Changeability assessment relies on an effective tradespace exploration that provides a unified view of different system configurations and control policies. To support the design teams in exploring such tradespaces, this paper presents an approach combining Dynamic Programming (DP) and Reinforcement Learning (RL) for evaluating optimal control policies, illustrated through a wheel loader application. The underlying basis is that the overall task can be decomposed into several sub-tasks to be solved by DP or RL selectively. A control policy combined from these sub-tasks is presented along with an illustrative tradespace mapping system attributes. The results show that by combining the strengths of DP and RL, the proposed approach can be beneficial when exploring a wide range of solutions. It allows direct comparisons between configuration and control policy changes, which is crucial for effective changeability assessment. However, several limitations have been acknowledged and will be addressed in future studies.