A Comparative Analysis of Loosely and Tightly Coupled Accelerator Architectures for Machine Learning

With the rapid innovation of AI/ML workloads, accelerators have become an important part of the computing resources in datacenters. While characteristics such as performance, power, and efficiency are critical, developer velocity, which dictates how quickly a new workload can be deployed on an accelerator, is equally important. The accelerator architecture and properties it exposes to the higher-level programming model play a key role in determining the programming model and economic viability of the accelerators. We compare two classes of AI/ML accelerator architectures: an efficient loosely coupled accelerator already implemented in silicon and a tightly coupled accelerator with a traditional and user-friendly programming model. Our comparison shows that the performance of the design depends on the amount of hardware resources and not on how they are integrated, concluding that it is possible to architect accelerators in a much simpler and more compiler-friendly manner, while still maintaining the benefits of offload computing.