VectorWeaver: A Stage-Aware Automated Optimization Framework for LLM Inference on Edge Platform

The convergence of Large Language Models (LLMs) and the open-source RISC-V architecture creates significant opportunities for specialized AI hardware. However, achieving optimal LLM inference performance on RISC-V platforms is challenging due to the complex computational patterns of LLM operators and the architectural nuances of the RISC-V Vector (RVV) extension. Existing optimization strategies often rely on manual, expert-driven tuning, which is labor-intensive and not generalizable across diverse operators and hardware configurations. This article presents VectorWeaver, an automated framework to generate high-performance, stage-aware microkernels for LLM inference on RISC-V. VectorWeaver specifically addresses key performance bottlenecks in quantized models, such as memory discontinuities in vector dot-product operations, by employing a widen vectorization strategy. Guided by a stage-aware performance model, the framework systematically explores a rich optimization space encompassing loop unrolling, software pipelining, and mathematical approximations. The model generates distinct, highly-optimized kernels for the prefill and decode stages, selected at runtime via a dispatch mechanism. We evaluate VectorWeaver on the TH1520 and the Sophon SG2044 using Gemma3 and Qwen3 models with sizes ranging from 270M to 4B, where our generated microkernels achieve up to a 24x speedup over baseline implementations. For end-to-end evaluations, VectorWeaver delivers up to a 5.2x increase in inference throughput, demonstrating its effectiveness in automating the generation of high-performance code and advancing the state of LLM inference on the RISC-V ecosystem.