Two Numbers for a GPU: How Workload Size Affects Attainable Performance

Peak rates tell us how fast a graphics processing unit (GPU) can run, but not how large a job must be before it runs that fast. The roofline model, the usual tool for reasoning about GPU kernels, shares this gap. We address it with one more number per operation: the half-performance length n½, the job size at which a machine reaches half of its best rate r∞, a quantity introduced by Hockney and Jesshope for vector computers. We first derive a formula for machines that work in parallel rounds, as a GPU does: averaged over job sizes, n½ is the startup work plus half a round, n½ = r∞T₀ + (n_w − 1)/2, and it recovers each of their machine classes. We then measure both parameters on an NVIDIA H100 SXM, for memory traffic and two shapes of matrix multiplication in BF16 (bfloat16), in three timing regimes: called from PyTorch, replayed from a CUDA Graph, and amortized over many copies in one graph. Called from PyTorch, square matrix products reach r∞ = 810 TFLOPS with n½ = 13.7 GFLOP, and memory traffic reaches 3,089 GB/s with n½ = 34 MB. The fixed cost depends strongly on how the work is issued: amortized over many copies, n½ falls to 3.2 GFLOP and 4.8 MB, while r∞ changes by at most 4%. The two numbers predict sizes left out of the fit with a mean error of 4.8% for square products and 1.5% for memory traffic outside the on-chip cache, and richer models with up to four parameters do at most about one point better. Three sessions on one machine agree within 3.5%, but a second H100 reached a 15% lower r∞ for matrix products, so we recommend measuring the parameters on the machine of interest. Finally, the model shows what a roofline time estimate misses: on the fitted curve, square products between about 780 and 1,170 in size count as compute-bound, yet run below half the compute rate even when launch cost is amortized.