K2V: Generative Keyframing For Physically Plausible Video Generation
Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.
Authors
- Roberto Cipolla (ORCID: https://orcid.org/0000-0002-8999-2151)
- Daniel Olmeda Reino
- Ayush Tewari (ORCID: https://orcid.org/0000-0002-3805-4421)
- Felix O'Mahony
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.48550/arxiv.2611.17324
- Primary Topic
- Generative Adversarial Networks and Image Synthesis
- Type
- preprint