An LLVM-Based HW/SW Co-Design Framework for Overlapped Chunked Execution of Single-Buffer FPGA Accelerators

Edge AI accelerators frequently adopt a single-buffer structure to minimize on-chip memory, but such accelerators cannot accept the next block of input while the current block is still being processed, which serializes data transfer, computation, and result retrieval. This paper presents a hardware/software co-design framework that restores system-level overlap for chains of single-buffer accelerators connected through AXI DMA. On the hardware side, a Hardware Performance Manager (HPM) monitors the AXI4-Stream handshakes between the DMA and the accelerator and coordinates the output handshake while a new input packet is starting. On the software side, an LLVM Pass analyzes a sequential chunked offload loop with scalar-evolution (SCEV) expressions, verifies that consecutive accelerator stages form a producer–consumer chain, and replaces the loop with a call to a pipelined runtime. The Pass additionally moves whole-buffer cache maintenance in front of and behind the loop into per-chunk operations that the runtime performs while the accelerators are busy. On a Zybo Z7-20 board with a CORDIC–GEMM chain, the transformed program reduces the total execution time by 30.2–31.3% relative to sequential execution for 201 K–4.8 M input words, and all 1040 measured runs, which also include single-stage loops and a loop with a shorter final chunk, reproduce a bit-exact software reference. An exhaustive chunk-size sweep shows that the default chunk size lies within 0.44% of the best fixed chunk size for every workload size.