Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap

When introducing new operators into Large Language Model (LLM) workloads, a key question is whether fusing the operator into a dependent kernel or overlapping its execution with another independent operator will yield better hardware efficiency. In this work, we analyze these optimization options for the Dropout operator, evaluating fine-grained architecture resource constraints and performance impacts beyond traditional compute or memory metrics. When enabled, Dropout is likely to dramatically impact the performance of end-to-end LLM training. The state-of-the-art optimization is to fuse Dropout into the Flash-Attention kernel; however, fusion fails to fully hide the latency of Random Number Generation (RNG), Dropout's main runtime contributor, due to shared lower-level architecture bottlenecks which are often overlooked. We propose overlapping RNG with preceding independent operators, specifically GEneral Matrix Multiplication (GEMM) layers, to hide RNG latency and improve end-to-end performance. RNG and GEMM have distinct resource requirements and hardware bottlenecks, so they can run together without compromising each other's performance. Our analytical performance model, validated on GH100 GPUs with FP8 precision, shows 1.26x speedup for overlapping RNG and GEMM layers over sequential execution and 1.22x over state-of-the-art fusion approach on Llama3 for a single Transformer block. Our methodology can be generalized to determine when to overlap or fuse new LLM operators across various architectures, models, and configurations.