LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping
Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from $17$ object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.
Authors
- Apoorva Vashisth (ORCID: https://orcid.org/0009-0005-3741-9053)
- Manav Kulshrestha (ORCID: https://orcid.org/0009-0004-3266-8328)
- Shreyans Jain (ORCID: https://orcid.org/0000-0001-7338-1666)
- Aniket Bera (ORCID: https://orcid.org/0000-0002-0182-6985)
- Damon Conover
- Ankush Gupta (ORCID: https://orcid.org/0009-0006-0596-5436)
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.48550/arxiv.2611.23858
- Primary Topic
- Robot Manipulation and Learning
- Type
- preprint