Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation
Robot reward modeling faces two challenges: generalizing across tasks and accurately tracking progress in long-horizon manipulation. General-purpose reward models demonstrate broad task transfer, but transfer from short-task pretraining to long-horizon tasks remains underexplored. Long-horizon progress models often rely on task-specific subtask classifiers or fixed action vocabularies, limiting the reuse of learned progress knowledge when adapting to new tasks. To address these challenges, we introduce Grana, a vision-language robot reward model that learns to identify unfinished subtasks and estimate their continuous progress using open-vocabulary subtask descriptions. We train a language-conditioned model to estimate progress in short-horizon tasks and individual subtasks of long-horizon tasks. For long-horizon tasks, we combine the selected subtask's predicted progress with credit for preceding completed subtasks to estimate whole-task progress. General progress prediction requires training the model across diverse tasks with progress supervision that accounts for imperfect execution. Elapsed-time proxies can misrepresent progress when relative subtask durations vary, or failures reverse local progress. We therefore introduce an annotation protocol that converts sparse subtask boundaries and action-quality annotations into dense progress targets, without requiring annotators to assign numerical progress scores. Pretraining with this supervision on 200 diverse, mostly short-horizon manipulation tasks enables zero-shot reward prediction on unseen tasks and improves progress prediction after adaptation to multi-minute, long-horizon tasks. Experiments demonstrate (i) generalizable reward prediction, including adaptation to five long-horizon tasks, (ii) benefits from pretraining and composing subtask progress estimates, and (iii) improved downstream policy performance.
Authors
- Ruijie Zheng (ORCID: https://orcid.org/0009-0004-2217-1318)
- Fengyuan Hu (ORCID: https://orcid.org/0009-0006-9785-0934)
- Haonan Duan (ORCID: https://orcid.org/0000-0002-3608-4467)
- Jiasheng Gu (ORCID: https://orcid.org/0009-0004-2678-1607)
- Radha Poovendran (ORCID: https://orcid.org/0000-0003-0269-8097)
- Kaiyuan Zheng (ORCID: https://orcid.org/0000-0002-5727-493X)
- Kun Lei (ORCID: https://orcid.org/0000-0003-0758-1488)
- Fangqi Zhu
- Nan Zhu (ORCID: https://orcid.org/0000-0002-8073-149X)
- Yinzhen Xu
- Guanzhi Wang (ORCID: https://orcid.org/0009-0004-1921-1796)
- Yuke Zhu
- Linxi Fan
- Shenyuan Gao
- Zi-ang Cao
- Zhe Zhang
- Kevin Lin
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.48550/arxiv.2611.24926
- Primary Topic
- Robot Manipulation and Learning
- Type
- preprint