Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation

Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.

Authors

Institutions

Publication Details

Journal
Sensors
Published
2026-11-18
DOI
https://doi.org/10.3390/s26227292
Primary Topic
Robot Manipulation and Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation

Xuancen Liu, Cheng Fu, Houyuan Xiao, Wenqing Zhang et al.
Sensors
Robot Manipulation and Learning
article

Learning Where to Grasp and Place: Role-Specific Attention for Language-Conditioned Manipulation

Xuancen Liu, Cheng Fu, Houyuan Xiao, Wenqing Zhang, Shengli Wang, Bin Wang
article en

Abstract

Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.

SensorsVol. 26(22)
China Aerodynamics Research and Development Center (CN)
Robot Manipulation and Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon