Motion-Aware Reward Modeling for Instruction-Guided Video Editing
Reward models for instruction-guided video editing can favor temporal consistency at the expense of motion. We study this failure, termed \emph{motion collapse}, using controlled motion-suppression variants of real edits. On VE-Bench, the evaluated composite reward prefers a motion-suppressed candidate in $97\%$ of cases. Learned video rewards and a VLM judge also struggle with mild suppression, while latent optimization and reward-guided fine-tuning exploit this weakness to increase reward as motion declines. A linear probe of our base reward backbone performs near chance on mild suppression, motivating an explicit inter-frame dynamics input. We propose a motion-aware reward model that combines an RGB frame-difference stream with preference training against motion-suppressed and fabricated-motion negatives. A distillation anchor preserves alignment with human quality judgments. On the VE-Bench motion ladder, our model reduces the rate of selecting motion-suppressed candidates to $0\%$ while achieving a Spearman correlation of $0.57$ with human quality ratings. It also detects unseen suppression variants and preserves motion during reward-guided fine-tuning on two editing pipelines. These results support explicitly modeling motion in rewards for video editing.
Authors
- Zhengzhong Tu (ORCID: https://orcid.org/0000-0002-7594-2292)
- Franck Dernoncourt (ORCID: https://orcid.org/0000-0002-1119-1346)
- Tyler Derr (ORCID: https://orcid.org/0000-0002-0080-5998)
- Viet Dac Lai (ORCID: https://orcid.org/0009-0008-1651-4619)
- Samyadeep Basu (ORCID: https://orcid.org/0009-0007-4021-0239)
- Bo Ni (ORCID: https://orcid.org/0009-0000-3795-4148)
- Thien Huu Nguyen
- Seunghyun Yoon
- Xiangbo Gao
- Yue Zhao
- Li Li
- Gang Wu
- Jing Shi
- Yu Wang
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.48550/arxiv.2611.22495
- Primary Topic
- Generative Adversarial Networks and Image Synthesis
- Type
- preprint