Multimodal tactile–posture capture enables physically grounded annotation of human demonstrations

Abstract Learning dexterous manipulation from human demonstrations requires records of physical interactions that video alone cannot fully capture. Here we present tactile-motion capture (TaMoCap), a portable physical interaction tracking system that records multimodal tactile–posture information during natural human demonstrations, reconstructs physical interaction states of human-object interaction, and converts them into structured semantic annotations. The system integrates a smart textile glove with distributed three-axis force, triboelectric and strain sensors. The synchronized multimodal sensing streams are processed to reconstruct three-axis contact forces, strain-derived joint-angle estimates, and triboelectricity-based contact-transition states, yielding physically interpretable representations that characterize sustained loading, transient interaction events, and hand posture during natural manipulation. A tactile semantic translation framework integrates synchronized visual observations, reconstructed physical interaction states, and task context to generate action annotations, which are verified against multimodal evidence for physical consistency and source traceability. In a descriptive evaluation of four selected massage cases, the full visual–tactile–semantic input achieves a mean composite score of 94.75/100 in model-based assessment, 33.5 points higher than the video-only input. In a long-horizon noodle-cooking demonstration, TaMoCap preserves temporal alignment between multimodal sensing evidence and structured semantic annotations throughout the action sequence. TaMoCap provides a systematic approach for constructing physically grounded human demonstration records for tactile-enhanced embodied agents and robotic manipulation systems.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11217490/v1
Primary Topic
Robot Manipulation and Learning
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Multimodal tactile–posture capture enables physically grounded annotation of human demonstrations

Herong Wang, Gaojie Chen, Ning Li, Yue Xiao et al.
2 links
Research Square
Robot Manipulation and Learning
preprint

Multimodal tactile–posture capture enables physically grounded annotation of human demonstrations

Herong Wang, Gaojie Chen, Ning Li, Yue Xiao, Yuzhou Wang, Xiaojian Rao, Jindong Tian, Yong Tian, Lin Fan, Yiyang Li, Weiguan Zhang
preprint en

Abstract

Abstract Learning dexterous manipulation from human demonstrations requires records of physical interactions that video alone cannot fully capture. Here we present tactile-motion capture (TaMoCap), a portable physical interaction tracking system that records multimodal tactile–posture information during natural human demonstrations, reconstructs physical interaction states of human-object interaction, and converts them into structured semantic annotations. The system integrates a smart textile glove with distributed three-axis force, triboelectric and strain sensors. The synchronized multimodal sensing streams are processed to reconstruct three-axis contact forces, strain-derived joint-angle estimates, and triboelectricity-based contact-transition states, yielding physically interpretable representations that characterize sustained loading, transient interaction events, and hand posture during natural manipulation. A tactile semantic translation framework integrates synchronized visual observations, reconstructed physical interaction states, and task context to generate action annotations, which are verified against multimodal evidence for physical consistency and source traceability. In a descriptive evaluation of four selected massage cases, the full visual–tactile–semantic input achieves a mean composite score of 94.75/100 in model-based assessment, 33.5 points higher than the video-only input. In a long-horizon noodle-cooking demonstration, TaMoCap preserves temporal alignment between multimodal sensing evidence and structured semantic annotations throughout the action sequence. TaMoCap provides a systematic approach for constructing physically grounded human demonstration records for tactile-enhanced embodied agents and robotic manipulation systems.

Research Square
Sun Yat-sen University (CN), Shenzhen University (CN), Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen) (CN)
Sun Yat-sen University, Basic and Applied Basic Research Foundation of Guangdong Province
Robot Manipulation and Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon