InTRRA: Inter-Task Resource-Repurposing Accelerator for Efficient Transformer Inference on FPGAs

The rise of deep neural networks (DNNs) has driven a boom in AI services, accompanied by increasing demands for computing power and memory. One of the most widely used DNNs is the transformer model. Unlike convolutional neural networks (CNNs), transformers involve more complex computations and generate larger intermediate data, which pose significant challenges for computation efficiency and off-chip memory access overhead in large language model (LLM) accelerator designs. Unfortunately, existing LLM accelerators have fixed execution patterns (either dataflow or sequential), resulting in either low computational efficiency due to pipeline stalls or frequent off-chip memory accesses to manage large intermediate results. This ultimately leads to high single-batch inference latency.