Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms

Cloud platforms have deployed hardware accelerators like neural processing units (NPUs) for machine learning (ML) inference services. To maximize resource utilization while ensuring quality of service, a natural approach is to virtualize NPUs for efficient resource sharing for multi-tenant ML services. However, virtualizing NPUs is challenging. This is not only due to the lack of system abstraction support for NPU hardware, but also due to the lack of architectural and ISA support for fine-grained operator scheduling. This article presents Neu10, an NPU virtualization framework consisting of (1) an abstraction called vNPU for fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU allocator that enables pay-as-you-go pricing and flexible vNPU-to-pNPU mappings; and (3) an ISA extension of modern NPU architecture for fine-grained tensor operator scheduling for multiple vNPUs. We evaluate Neu10 with a production-level simulator to demonstrate its benefits over state-of-the-art NPU sharing approaches.