Benchmarking the performance of GPT-2 type applications on GPU-accelerated computing resources

In this report we look at eight GPU-accelerated systems representing across-section of the available Tier 2 HPC systems in the UK. These includea mixture of GPU-accelerated platforms from NVIDIA, AMD and Intel. For each system, we perform a set of benchmarking experiments by training a GPT-2 model using a mixture of parameters and hyperparameters. These include model size, number of GPUs, floating point data type, training datasize, distribution strategies and batch size. Our interest is in performance measured by time taken to complete an epoch of training rather than convergence speed. We also measure memory usage. The overall aim is to compare systems in order to provide researchers intending to perform AI training with some benchmarks for what to expect in terms of training speed for a modelin a non-optimised real-world scenario.