Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates

Objective. Neural decoders for dexterous finger movements are limited by the accuracy with which ground truth movement is measured. In non-human primates (NHPs), hand kinematics are typically recorded with instrumentation that constrains movement dimensionality. We evaluated whether a markerless camera system could replace conventional methods for tracking movements in real time. Approach. We developed Freebie, a real-time markerless system for tracking NHP finger movements. Keypoints detected online were used in place of manipulandum bend sensor signals. Two NHPs performed tasks by flexing fingers to move a virtual hand into target zones. The virtual hand was directly controlled using bend sensors or Freebie-derived kinematics, or indirectly controlled through a Kalman filter (KF) decoding kinematics from forearm electromyographic (EMG) activity. We quantified Freebie's accuracy relative to manual labels, agreement with bend sensor measurements, and latencies. We assessed online performance under Freebie control and compared EMG KFs trained using Freebie-derived labels with those trained using bend-sensor-derived labels. Main Results. We observed a median end-to-end latency of 15.8 ms at a 30 FPS camera acquisition rate, and 16.2 ms at 60 FPS. Freebie labels closely tracked bend sensor positions (median |r|=0.96). Median keypoint errors were 1.7 to 6.5 pixels, below the 20-pixel width of the smallest tracked finger. Direct Freebie control did not differ from bend sensor control on five of six online metrics; path efficiency was higher with Freebie. EMG KFs trained on Freebie-derived labels performed similarly to those trained on bend sensor labels online. When evaluated offline, models trained with Freebie had lower error. Significance. These findings indicate that bend sensors are not required to obtain finger labels for training a decoder: markerless keypoints alone provided effective labels. This is an important first step towards real-time tracking of more complex movements, beyond what could be easily measured with a manipulandum.

Authors

Institutions

Publication Details

Journal
bioRxiv (Cold Spring Harbor Laboratory)
Published
2026-11-12
DOI
https://doi.org/10.64898/2026.11.11.765959
Primary Topic
Motor Control and Adaptation
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates

Cynthia A. Chestek, Matthew J. Mender, Hisham Temmar, Ayobami L. Ward et al.
bioRxiv (Cold Spring Harbor Laboratory)
Motor Control and Adaptation
preprint

Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates

Cynthia A. Chestek, Matthew J. Mender, Hisham Temmar, Ayobami L. Ward, Anne Draelos, Madison M. Kelberman, Theodore A. Kung, Matteo Ceradini, Miranda G. Copenhaver, Olivia Lee, Jake C Joseph, Jesse DeMatteo, Lucy Liu
preprint en

Abstract

Objective. Neural decoders for dexterous finger movements are limited by the accuracy with which ground truth movement is measured. In non-human primates (NHPs), hand kinematics are typically recorded with instrumentation that constrains movement dimensionality. We evaluated whether a markerless camera system could replace conventional methods for tracking movements in real time. Approach. We developed Freebie, a real-time markerless system for tracking NHP finger movements. Keypoints detected online were used in place of manipulandum bend sensor signals. Two NHPs performed tasks by flexing fingers to move a virtual hand into target zones. The virtual hand was directly controlled using bend sensors or Freebie-derived kinematics, or indirectly controlled through a Kalman filter (KF) decoding kinematics from forearm electromyographic (EMG) activity. We quantified Freebie's accuracy relative to manual labels, agreement with bend sensor measurements, and latencies. We assessed online performance under Freebie control and compared EMG KFs trained using Freebie-derived labels with those trained using bend-sensor-derived labels. Main Results. We observed a median end-to-end latency of 15.8 ms at a 30 FPS camera acquisition rate, and 16.2 ms at 60 FPS. Freebie labels closely tracked bend sensor positions (median |r|=0.96). Median keypoint errors were 1.7 to 6.5 pixels, below the 20-pixel width of the smallest tracked finger. Direct Freebie control did not differ from bend sensor control on five of six online metrics; path efficiency was higher with Freebie. EMG KFs trained on Freebie-derived labels performed similarly to those trained on bend sensor labels online. When evaluated offline, models trained with Freebie had lower error. Significance. These findings indicate that bend sensors are not required to obtain finger labels for training a decoder: markerless keypoints alone provided effective labels. This is an important first step towards real-time tracking of more complex movements, beyond what could be easily measured with a manipulandum.

bioRxiv (Cold Spring Harbor Laboratory)
Scuola Superiore Sant'Anna (IT), University of Michigan (US)
National Science Foundation, Burroughs Wellcome Fund, Alfred P. Sloan Foundation, National Institute of General Medical Sciences
Motor Control and Adaptation
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon