Towards Energy-Efficient Llama2 Architecture on Embedded FPGAs

Large language models (LLMs) have shown immense potential for applications in information retrieval and knowledge management, but their computational and memory demands pose challenges for resource-constrained devices. In response, this work introduces an FPGA-based accelerator designed to improve LLM inference performance on embedded devices. We leverage quantization techniques, asynchronous computation, and a fully-pipelined accelerator to enhance efficiency. Our empirical evaluations, conducted using the TinyLlama 1.1B model on a Xilinx ZCU102 platform, demonstrate a 14.3-15.8x speedup and a 6.1x energy efficiency improvement over running exclusively on the ZCU102 processing system (PS).