Co-design of a TinyLLM using Programmable Logic and Software on an FPGA

Large language models (LLMs) provide unique scalable solutions to many natural language processing tasks such as translation, text generation, and content summarization. However, they require significant computing power to achieve high token throughput, which is costly. This work explores low-cost, low-power solutions for implementing LLMs in embedded hardware, describing the design space exploration (DSE) performed and the resulting system architecture. Based on the DSE, we demonstrate an adapted implementation of a quantized LLM using programmable logic (PL) and programmable software (PS) on a ZYNQ FPGA and ARM hybrid system. This TinyLLM retains the ability to produce cohesive text, generating 3.78 tokens/s with an observed 37x speedup over a CPU-based implementation on the same hardware while consuming just under 1.5W of power.