Understanding Oversubscribed Memory Management for Deep Learning Training

This paper analyzes the performance impact of unified virtual memory (UVM) while running large deep learning (DL) workloads on a GPU with limited memory capacity. Due to the page fault handling overhead of UVM, DL frameworks have not officially supported UVM. However, given the rising GPU price, the global semiconductor shortages, and the ever-increasing DL model size, we cannot optimistically rely on adding more GPUs for DL processing. Various other solutions such as compression and quantization reduce memory footprint at the cost of performance and accuracy. Using UVM with memory oversubscription could tackle such concerns. However, UVM has not been actively leveraged in DL computing due to its performance overhead. In this study, we investigate the performance impact of UVM for DL computing to better understand the benefits and limitations. Our results show that while UVM enables training large models beyond GPU capacity, its effectiveness depends on the interplay between oversubscription factors and memory management strategies. We find that PCA significantly mitigates page fault overhead, improving performance at moderate oversubscription levels, but also increases migration traffic. These findings highlight the potential of integrating UVM with advanced memory management strategies to optimize DL workloads on limited-memory GPUs.