EVBench: A Comprehensive Benchmark of Video Generative Models for E-commerce Scenarios

Assessing the quality of videos generated by video generation models plays an important role in improving the capability of these models. Existing video generation benchmarks like VBench and EvalCrafter mainly focus on general aspects of video quality and video-text consistency, but they could be ineffective for unique requirements of e-commerce video generation, such as the consistency of fine-grained product categories and details, the usage of products, camera movements, etc. To solve this issue, we introduce EVBench , a comprehensive E -commerce V ideo Bench mark to evaluate the capability of video generation models for e-commerce scenarios. Our EVBench contains two parts, i.e., EVBench-T2V and EVBench-I2V, to evaluate the text-to-video models and image-to-video models, respectively. EVBench-T2V contains 300 structured prompts covering 300 fine-grained product categories and various backgrounds, while EVBench-I2V has 120 image-prompt pairs of 120 categories with diverse instructions like camera movements and usage scenarios. Moreover, a set of novel evaluation metrics is designed to quantitatively measure the capability of video generation models from 10 dimensions. Furthermore, we conduct the human preference annotation to validate the alignment between our benchmark and human perception. Finally, based on EVBench, we comprehensively evaluate 15 mainstream open-source and closed-source models to discover their capabilities and provide valuable insights for e-commerce video generation.