Benchmarking Mobile Deep Learning Software

Deploying deep learning (DL) on mobile devices has become increasingly prevalent. DL software libraries are crucial for efficient on-device inference, alongside algorithms and hardware. However, there has been limited understanding on the performance of modern DL libraries. We fill this gap by benchmarking 6 popular DL libraries and 15 diverse models across 10 mobile devices, which reveal an unsatisfactory landscape of mobile DL: their performance is highly disparate and fragmented across different models and hardware, and the impacts often surpass algorithm or hardware optimizations, such as model quantization and GPU/NPU-based computing. Finally, we provide practical implications for stakeholders in the DL library ecosystem, and envision a more ambitious picture of future mobile AI landscape in the LLM era.