Train Everything, Everywhere, All at Once: A Version Control Convention for Training Multiple Deep Learning Models

Machine learning (ML) researchers specializing in neural networks and deep learning (DL) often perform numerous experiments, and it is challenging to track and monitor the performance of different models efficiently. An ML team must have an efficient method to track performance and label each experiment accordingly. Borrowing version control concepts from software engineering, we formulated our version control convention (VCC) and applied it to two DL projects, both of which are published in SCOPUS Q1 and Q2 journals. We use the VCC label (e.g., “V.XX.YY.ZZ”) for a YAML-based network configuration setup and incorporate a factory software design pattern to facilitate the creation of different network architectures based on a given version label. Hyperparameters and training configurations are also set based on the given version label. Setting up batch sizes that could fit a given GPU memory is also tied to the version label. We discuss our general DL development practice and experiences of applying our VCC. Using our proposed convention and given a particular task, we can train everything (all possible DL models) and train everywhere (all available workstations with different GPU configurations) all at once (simultaneously).