BBP-DNS: Batch-Block Parallelism and Dual-NoC Scheduling for Accelerating GPT on Edge Devices

Transformer-based GPT models in artificial intelligence have exhibited remarkable performance advantages across generative tasks. However, edge-side deployment of GPT models faces significant challenges due to the limited memory and computational resources of edge devices. The lack of specialized compiler toolchains often necessitates manual compilation, resulting in complicated deployment processes. To address these challenges, we propose a Batch-Block Parallelization and Dual-NoC Scheduling algorithm(BBP-DNS) for edge devices, which automates tensor partitioning and batch scheduling to enhance data locality and reduce overheads in data storage, movement, and kernel scheduling, thereby improving inference speed. To mitigate hardware constraints in accelerator memory, bandwidth, and compute resources, we extend tensor parallelism with a fine-grained batch-level scheduling strategy. Additionally, we introduce a novel operator mapping methodology that automates accelerator data management, addressing the inefficiencies of traditional manual compilation workflow. Experimental results demonstrate BBP-DNS’s superior performance, achieving a 30x performance improvement of a GPT single-layer block on simulator of that evaluated by TVM and PyTorch on CPUs.