EffBench: A Holistic Efficiency Benchmark for CTR Models

Click-Through Rate (CTR) prediction is a critical task in recommender systems and online advertising, so CTR models design has been widely researched in both academia and industry. However, since the model parameters increase larger and their computations become more complex, the inefficiency severely hinders practical deployment and application. Especially, the lack of a unified benchmark for the computational efficiency of CTR models complicates matters. To address the issues, we propose a holistic efficiency benchmark for CTR models named EffBench. a) A comprehensive evaluation metric system is constructed for CTR models, including seven metrics to assess accuracy and strongly emphasize computational efficiency. b) EffBench benchmark is implemented based on industrial mainstream framework TensorFlow, to better align with real-world usage scenarios. c) An operator-level profiling tool in EffBench is designed to identify key performance bottlenecks. d) Based on the EffBench benchmark, we evaluate and analyze 30 representative models across three typical CTR task types on the three widely used real-world datasets. As the first benchmark aiming at improving CTR model efficiency, EffBench will facilitate researchers in CTR model evaluation, comparison, validation, and inspiring future model designs. Our benchmark source code is available at https://github.com/Yeedzhi/EffBench .