PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning
Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
Authors
- Lin Gan (ORCID: https://orcid.org/0000-0003-1696-7980)
- Boren Zheng
- Xuanyi Liu (ORCID: https://orcid.org/0000-0001-7542-3882)
- Danni Yu (ORCID: https://orcid.org/0000-0002-1868-109X)
- Pengcheng Wen
- Boyuan Chen
- Haofeng Wang
- Ruiqi Li
- Zhongyuan Wang
- Mingyuan Su
- Siyu Tao
- Haocheng Tang
- Tianzizhe Guo
- Yuhao Zhang
- Tiange Zhang
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.48550/arxiv.2611.16424
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- preprint