Benchmarking Deep Learning with Representative ONNX Subgraphs

Despite the ubiquity of deep learning (DL), there are a surprisingly limited number of accepted benchmarks for the task. The most common benchmark suite, MLPerf, tests the behavior of representative models from a variety of domains and showcases their peak performance. However, this benchmark does not provide insight into how a custom model will run on your system unless you happen to be using one of the models that are part of the benchmark (only one per domain). The insights yielded by MLPerf offer limited details into the behavior of the processor and do not show expected performance on a more general body of models. This research proposes, therefore, to leverage information about deep-learning models to support the creation of a data-driven subgraph benchmark. This research gathers key operational subgraphs from image-classification models and demonstrates that they can be used to characterize the performance of a system overall. This characterization offers a pathway inter-device comparison when the task is known but the device (and model specifics) are not.