Rule-Based Reinforcement Learning on FPGA for QoS-Aware Dynamic Frequency Scaling

To improve the system performance of multiprocessor system-on-chips (MPSoCs), modern processors have several built-in hardware features, such as prefetchers, which respond to short-term variations in processor load that occurs on a submillisecond scale. However, even the latest dynamic (voltage) frequency scaling governors using reinforcement learning (RL) are implemented in software and, thus, cannot take advantage of these variations. In this work, we propose a hardware RL agent, augmented with preemptive shielding and eligibility traces, to optimize the execution of deadline-bound quality-of-service (QoS) tasks in mixed-critical environments. We demonstrate the features of our algorithm in a hardware-in-the-loop simulation by running LLVM’s single-source benchmarks on SparcV8 processors. We also present our field-programmable gate array (FPGA) implementation with optimized resource usage and timing performance achieved through quantization and approximation.