LHS: LLM Assisted Efficient High-level Synthesis of Deep Learning Tasks

Deep learning tasks, especially those involving complex convolution neural networks (CNNs), are computationally intensive and pose significant challenges when implemented on hardware. Accelerating these tasks is critical for improving performance. High-level Synthesis (HLS) has the potential to automate the efficient hardware accelerator designs directly from high-level C/C++ specification of trained machine learning (ML) models. Traditional HLS tools cannot synthesize certain high-level constructs, which require manual intervention. Many source code optimizations and the selection of pragmas for HLS optimizations are crucial for generating efficient hardware accelerators with HLS. However, both of these tasks are mostly manual efforts. Recently, Large Language Models (LLMs) have shown remarkable capabilities in various generative tasks. In this work, we explore the application of LLMs to remove these manual efforts in adapting HLS for ML accelerator designs. Our framework called LLM-assisted HLS, i.e., LHS, uses LLMs to automate the resolution of synthesis issues, ensuring compatibility with HLS tools. Furthermore, our framework automates the source code modification and optimization selection through pragma insertion steps, which are crucial for optimizing the synthesized design. Our experimental results with LHS demonstrate a significant improvement in latency for deep learning tasks with underlying complex CNN models without much area overhead. Our LHS allows us to achieve up to 2690 × latency improvement. Promisingly, LHS performs better than the state-of-the-art ML accelerator design tool hls4ml in 4 out of 6 cases in the context of latency improvement at the expense of area overhead (i.e., performance to hardware gain). This work highlights the potential of LLMs to assist and accelerate the HLS process, thereby creating more efficient hardware implementation for deep learning models.