Introducing GPU Persistent Graphs for Time-sensitive Workflows

The emphasis on throughput in GPU design poses challenges when integrating them into time-sensitive applications. Recent advancements in GPU architectures and software have enabled the reduction of overhead and interference along the critical path through the use of advanced GPU mechanisms, such as persistent kernels. Despite these advancements, these methods often involve trade-offs in performance, flexibility, or portability. In response, we introduce our proposal of persistent graphs to enable a fully host-independent GPU pipeline scheduler. This concept is entirely modular, simple to implement and allows for online changes of workloads without impacting performance. We demonstrate the effectiveness of our approach through extensive benchmarking including an example of a highly constrained cyber-physical system: adaptive optics for astronomy. Our evaluation spans various state-of-the-art platforms, including A100 GPUs and embedded Jetson Orin SoCs. Notably, with our proposed implementation, response time variability never exceeds 10%, even on platform with significant CPU interference.