Seraph: A Performance-Cost Aware Tuner for Training Reinforcement Learning Model on Serverless Computing

Training a reinforcement learning model is critical for various AI tasks. However, determining the hardware resources required for training RL models is challenging due to the interaction between the CPU and GPU, and the variability that exists in the training. The problem becomes more challenging when deploying an RL training job on the cloud with serverless computing, as we should consider both the performance and cost of training RL models. Existing tuners, like Ray Tune, require users to provide a search space. It is both error-prone and unable to search the setup with the desired cost. We present Seraph, the first tuner for RL training that finds the hardware configuration with the best performance within a user-given cost boundary. Seraph explicitly models the performance of training by decomposing RL training and using a stochastic model to harness variability. Compared to Ray Tune, it finds the optimal with 71% tuning time reduction.