SPOC: Model Metrics and Inference Scalability

The growth of deep learning has increased the challenge of understanding model performance and scalability. Model metrics are coarse-grained values that help to capture the complexity of the model. In this research, we propose SPOC (size, parameters, operations, and Critical Datapath Length) as a suite of metrics that describe much of the complexity in deep-learning architecture and therefore help to characterize their scalability. We utilize small, specially structured “micro-models” to showcase overall scalability trends across four devices: the ARM Cortex-A76 embedded CPU, the AMD EPYC 9374F server CPU, the NVIDIA Jetson Orin Nano embedded GPU, and the NVIDIA A100 server GPU. Through these micro-models, we vary components of SPOC and highlight different scaling behavior across these classes of devices. Finally, we provide examples of how SPOC can be used with standard CNNs and MobileViT to capture expected and measured performance differences during inference. Through this study, we provide deep-learning practitioners with an overview of how metrics can guide our understanding of inference performance.