Evaluating Multi-Instance DNN Inferencing on Multiple Accelerators of an Edge Device

Edge devices like Nvidia Jetson have started to have multiple on-board accelerators such as GPU CUDA cores, Tensor Cores and Deep Learning Accelerators (DLA). Maximizing the DNN inferencing performance of such devices requires us to concurrently use these col-located hardware components, but this has not yet been studied. We analyze the performance of accelerators present in Jetson AGX Orin, both independently and concurrently, using multiple instances of ResNet50 model. We assess the effects of using different combinations of the components and varying batch sizes on the inference throughput and latency. Our results indicate that using CUDA Core with Tensor Cores offers a higher throughput, while using them in conjunction with DLAs reduces the benefits. This paves the way to explore more intelligent configurations to maximize the performance of edge platforms for AI workloads.