FastMatch: Enhancing Data Pipeline Efficiency for Accelerated Distributed Training

Training a deep learning model typically involves two interdependent stages: preprocessing input data on the CPU and training the model on the GPU. This process begins with the CPU performing resource-intensive I/O operations to load raw data, which it then processes and supplies to the GPU. This creates a typical provider-consumer relationship between the CPU and GPU. However, the inefficiency of I/O and CPU operations, in contrast to the high capabilities of GPUs, often turns preprocessing into a significant bottleneck. The imbalance between CPU and GPU resources leads to inefficiency, as the underutilized GPU cannot fully exploit its computational potential. To address this issue, we propose FastMatch, a system designed to selectively and precisely cache key data, thereby reducing the reliance on extensive I/O operations, and adjusting the CPU's performance to better align with the GPU's capabilities swiftly. FastMatch dynamically identifies important data and prompts the caching system to store it in fast DRAM. This cached data significantly enhances I/O performance while maintaining comparable accuracy levels. Additionally, FastMatch monitors and rectifies mismatches in CPU and GPU resources by tuning the number of preprocessing processes. This helps distribute preprocessing workloads more evenly, ensuring optimal utilization of both CPU and GPU resources. We implemented a prototype of FastMatch and successfully integrated it with TorchData. Our evaluations showed up to 2.2 × reduction in training time compared to vanilla PyTorch Dataloader.