Composing Robot Manipulation Skills from Linear Motion Codes

We study a compact action space for robot manipulation and a planner that composes skills in it. Each end-effector (EE) motion segment (reach, grasp, carry, push, ...) is stored in an object’s frame with rotations in tangent (log-map) coordinates, resampled to 50 phases and encoded by 16–24 PCA coefficients. Every code decodes to a valid SE(3) trajectory that the arm’s controller executes directly, held-out reconstruction is sub-millimetre, and a weighted sum of codes is the same weighted sum of motions. A 1.2M-parameter transformer, called once per segment with that segment’s instruction, predicts which object frame to use, the gripper event and the code, reading a library of training codes through attention restricted to the instruction. In ManiSkill3 simulation with one arm and state observations, it solves 87.1% of the test episodes of 13 trained tasks with a closed-loop insertion servo, and 91.6% when insertion segments also stop on slip (three seeds). On 537 fresh scenes of eight skill–object combinations never seen in training it solves 43.3% (57.9% with hand-written closed-loop finishes); one sentence describing several skills is expanded into a chain of known skills and solved 43% and 39% of the time on a phrasing set written before development. We report what did not work as carefully as what did. Blending codes, the property the space was built for, failed a pre-registered test; what made the planner generalize was storing each segment in the frame its end is fixed to. A learned goal reader trained on the library’s own sentences generalized worse than a small hand-written lexicon. And insertion failures we had attributed to contact turned out to be an open-loop motion that does not notice it is blocked: stopping a segment when the held object slips in the grasp, read from poses alone, raises peg insertion from 18.8% to 66.5%. All results are in simulation with a single robot; the closed-loop finishes are hand-written and reported separately.