Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation
Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.
Authors
- Xuancen Liu (ORCID: https://orcid.org/0000-0002-6956-4141)
- Cheng Fu (ORCID: https://orcid.org/0000-0002-5367-9905)
- Houyuan Xiao (ORCID: https://orcid.org/0009-0007-7930-1111)
- Wenqing Zhang
- Shengli Wang
- Bin Wang
Institutions
- China Aerodynamics Research and Development Center (CN)
Publication Details
- Journal
- Sensors
- Published
- 2026-11-18
- DOI
- https://doi.org/10.3390/s26227292
- Primary Topic
- Robot Manipulation and Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00