Temporal Preference-Consistent Multi-Timescale Credit Assignment for Digital Twin-Enabled Reinforcement Learning in Smart Manufacturing

Reinforcement learning agents deployed in digital twin-enabled smart manufacturing environments frequently encounter non-stationary operational conditions and multi-objective trade-offs that conventional static reward formulations cannot adequately address. We propose a temporal preference-consistent multi-timescale credit assignment framework that fundamentally restructures how reward signals are generated and how policy updates are performed within the standard deep reinforcement learning loop. A recurrent preference encoder, regularized for temporal smoothness, dynamically generates context-dependent preference vectors from operational metadata such as energy prices, production queue status, and machine health indicators, thereby preventing abrupt shifts in the reward landscape. Furthermore, a drift detection mechanism monitors discrepancies between digital twin predictions and observed physical states, triggering preference re-anchoring when regime changes occur. To address delayed consequences of control actions, we employ dual eligibility trace streams that separately capture immediate actuator effects and long-term throughput or quality outcomes. Counterfactual digital twin rollouts refine advantage estimation by simulating alternative trajectories under perturbed dynamics, hence improving robustness against sim-to-real discrepancies. Additionally, a prioritized replay strategy that considers both temporal difference error and preference novelty, combined with a temporal consistency loss during mini-batch training, stabilizes policy learning across evolving operational contexts. The proposed method is expected to yield more adaptive and stable control policies in real-time manufacturing settings, particularly where energy efficiency, product quality, and throughput must be balanced under continuously changing conditions.