LayoutFusion: A Data-Layout-Focused Operator Fusion Framework

Despite the increasingly memory-intensive nature of deep neural networks, mainstream deep learning compilers still fail to fully fuse memory-intensive operators, particularly data movement operations, during computational graph optimization. This is because existing approaches overlook the impact of changes in tensor data layouts on tile configuration selection, leading to the inclusion of inefficient tile configurations during operator generation. In this paper, we develop an operator fusion framework, LayoutFusion, approaching the problem from two key perspectives. For the tile configuration generation strategy, this paper introduces a tile configuration filtering mechanism based on sub-tensor data layout. For fusion group building, this paper proposes a two-stage algorithm that combines operator type classification with a non-greedy fusion strategy. On the GeForce RTX 3090 GPU platform, we evaluate LayoutFusion on 5 DNN models. The results show that our new constraint rule can filter out an average of 37% tile configurations. For end-toend latency, the models deployed with LayoutFusion achieve up to a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5.55 \times$</tex> speedup compared to the baseline.