UltraFormer: An Efficient Transformer for FPGAs

Transformer-based large language models (LLMs) have exploded in popularity in both industrial and academic settings, seeing rapid progress in model quality over the last few years and unprecedented levels of adoption worldwide. Acceleration of these transformer architectures and their efficient alternatives [1], [2] has surged in relevance to meet the aforementioned efficiency needs. Field programmable gate arrays (FPGAs) provide the ability to implement highly specialized accelerators for these transformer architectures. Recent work [3], [4] has demonstrated benefits but also unique challenges posed by FPGA-based solutions. We posit existing FPGA solutions are attempting to accelerate an algorithm that is ultimately a poor fit, resulting in avoidable inefficiencies. We address this with our proposed ultra-efficient architecture UltraFormer and design it with the challenges facing FPGA acceleration solutions in mind. This work makes a number of critical contributions: •We propose an efficient transformer architecture featuring the first-ever combination of hybrid linear attention and extremely low-precision linear projections. These features contribute to ~10x compression of the model weights and 13.2x key-value (KV) cache compression in long-context scenarios compared to Llama3 architectures. •We synthesize, simulate, and deploy UltraFormer via HLS flows [5] targeting enterprise-level Xilinx FPGAs. This represents the first time that an efficient attention mechanism featuring linear attention has been synthesized for FPGA deployment and the first time that BitNet b1.58 linear projections [6] have been synthesized or simulated for custom hardware. •We demonstrate that UltraFormer produces notable throughput and energy efficiency improvements. When deploying UltraFormer on a Xilinx VCK5000, we see up ro a 4.4x reduction in latency and a 3.2x reduction in power consumption compared to Llama3 running on a NVIDIA A100 GPU.