A Water-Flow Analogy for Teaching Data Reuse and Memory Hierarchies

Deep Learning kernels, such as general matrix multiplications (GEMMs), exhibit high data-reuse or operations per byte that help in improving their performance on hardware. Yet young architects often struggle to internalize the benefits of data reuse. In this work, we introduce an intuitive water-flow analogy for a simple memory-compute model to clarify how bandwidth, on-chip memory, data reuse and compute throughput interact to determine GEMM performance. In this analogy, DRAM is the tank; the memory controller is a tap with limited flow and latency; SRAM is the bucket; compute units are the dish-washer. We also illustrate the role of data reuse through worked out examples, showing compute- and memory- bound scenarios in roofline performance model, as well as latency-bound scenario. By grounding architectural ideas in physical world scenario, our water-flow analogy helps in longer retention of concepts related to data reuse and memory hierarchy. We also provide ready-to-use slides for its easy adoption in courses1.