Surgical Intelligence Labs: Collecting First-Person Surgical Data with a Smartphone to Train Autonomous Robots

Abstract Background. Robot foundation models learn from large, diverse data, and human video is becoming a powerful way to supply it. Surgery has almost none beyond console video, yet the hands of surgeons, assistants and scrub technologists, seen from their own eyes, could be the main source to train autonomous robots in the operating room (OR). Methods. We built Surgical Intelligence Labs, a smartphone app that turns a clinician's iPhone into a first-person (egocentric) data-collection device. A head-mounted phone records tasks from operative field and back-table setup to console and microscope views, saving each frame with its 6-degree-of-freedom (6-DoF) head pose from Apple's ARKit and 50 Hz device motion. Hands are tracked on the phone (21 key points each), lifted to three dimensions by fitting an articulated hand model, and retargeted to a Unitree G1 + humanoid with five-finger Wuji Hand 2 hands and to ASIA, our bimanual scrub-technician robot. OR recording requires institutional approval, consent and on-device de-identification. Results. On a 63-s suturing exercise recorded with a head-mounted iPhone 17, ARKit tracked the head in all 1,877 video frames, with the phone a mean 55° below the horizon, inside the framing guide's target band. Both hands were tracked on the phone in 97% of sampled frames, and the 3D hand fit was kept for 99% of hand observations. Position-based retargeting to five-finger Wuji Hand 2 hands kept thumb-to-fingertip gaps in pinches and grips within 2 mm of the fitted human gaps (medians). Inverse kinematics solved all 300 G1 + and 200 WidowX test poses (median errors 1.1 and 0.35 mm). Conclusions. A consumer smartphone can capture first-person video, head pose, and hand motion that can be mapped to actions humanoid and bimanual robots can execute. These findings demonstrate feasibility; they do not yet establish dataset quality or robot performance. Data collection, training, and deployment are the next steps.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11654389/v1
Primary Topic
Surgical Simulation and Training
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Surgical Intelligence Labs: Collecting First-Person Surgical Data with a Smartphone to Train Autonomous Robots

Joao Bombardelli, Mustafa Khan
2 links
Research Square
Surgical Simulation and Training
preprint

Surgical Intelligence Labs: Collecting First-Person Surgical Data with a Smartphone to Train Autonomous Robots

Joao Bombardelli, Mustafa Khan
preprint en

Abstract

Abstract Background. Robot foundation models learn from large, diverse data, and human video is becoming a powerful way to supply it. Surgery has almost none beyond console video, yet the hands of surgeons, assistants and scrub technologists, seen from their own eyes, could be the main source to train autonomous robots in the operating room (OR). Methods. We built Surgical Intelligence Labs, a smartphone app that turns a clinician's iPhone into a first-person (egocentric) data-collection device. A head-mounted phone records tasks from operative field and back-table setup to console and microscope views, saving each frame with its 6-degree-of-freedom (6-DoF) head pose from Apple's ARKit and 50 Hz device motion. Hands are tracked on the phone (21 key points each), lifted to three dimensions by fitting an articulated hand model, and retargeted to a Unitree G1 + humanoid with five-finger Wuji Hand 2 hands and to ASIA, our bimanual scrub-technician robot. OR recording requires institutional approval, consent and on-device de-identification. Results. On a 63-s suturing exercise recorded with a head-mounted iPhone 17, ARKit tracked the head in all 1,877 video frames, with the phone a mean 55° below the horizon, inside the framing guide's target band. Both hands were tracked on the phone in 97% of sampled frames, and the 3D hand fit was kept for 99% of hand observations. Position-based retargeting to five-finger Wuji Hand 2 hands kept thumb-to-fingertip gaps in pinches and grips within 2 mm of the fitted human gaps (medians). Inverse kinematics solved all 300 G1 + and 200 WidowX test poses (median errors 1.1 and 0.35 mm). Conclusions. A consumer smartphone can capture first-person video, head pose, and hand motion that can be mapped to actions humanoid and bimanual robots can execute. These findings demonstrate feasibility; they do not yet establish dataset quality or robot performance. Data collection, training, and deployment are the next steps.

Research Square
The University of Texas Health Science Center at Houston (US)
Good health and well-being
Surgical Simulation and Training
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon