Failure-Aware Adaptation for Free-Hand Interaction in Mixed Reality

Free-hand interaction in Mixed Reality (MR) can become difficult as the interaction technique, target geometry, and spatial context vary. We present FAIR-MR, a failure-aware adaptive interaction system that responds to unsuccessful interaction through geometric interventions. Two user studies characterize interaction difficulty across gestures and spatial conditions, and then compare Static, Precomputed, and Online Bandit strategies for selecting interventions. Interaction success varied across gestures, with target geometry affecting interaction techniques differently. The adaptation strategies produced distinct intervention and interaction patterns. Within Online Bandit, intervention estimates became increasingly differentiated with experience, and greater pre-decision differentiation was associated with subsequent recovery. However, this policy development did not produce higher aggregate success than the other strategies. These findings reveal how interaction conditions, intervention strategies, and learned policy behavior shape adaptive interaction. FAIR-MR contributes a system-level approach connecting interaction difficulty, intervention selection, interface changes, transition costs, and recovery.

Authors

Institutions

Publication Details

Journal
Multimodal Technologies and Interaction
Published
2026-11-13
DOI
https://doi.org/10.3390/mti10110122
Primary Topic
Interactive and Immersive Displays
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Failure-Aware Adaptation for Free-Hand Interaction in Mixed Reality

Arie E. Kaufman, Doris Gutiérrez
2 links
Multimodal Technologies and Interaction
Interactive and Immersive Displays
article

Failure-Aware Adaptation for Free-Hand Interaction in Mixed Reality

Arie E. Kaufman, Doris Gutiérrez
article en

Abstract

Free-hand interaction in Mixed Reality (MR) can become difficult as the interaction technique, target geometry, and spatial context vary. We present FAIR-MR, a failure-aware adaptive interaction system that responds to unsuccessful interaction through geometric interventions. Two user studies characterize interaction difficulty across gestures and spatial conditions, and then compare Static, Precomputed, and Online Bandit strategies for selecting interventions. Interaction success varied across gestures, with target geometry affecting interaction techniques differently. The adaptation strategies produced distinct intervention and interaction patterns. Within Online Bandit, intervention estimates became increasingly differentiated with experience, and greater pre-decision differentiation was associated with subsequent recovery. However, this policy development did not produce higher aggregate success than the other strategies. These findings reveal how interaction conditions, intervention strategies, and learned policy behavior shape adaptive interaction. FAIR-MR contributes a system-level approach connecting interaction difficulty, intervention selection, interface changes, transition costs, and recovery.

Multimodal Technologies and InteractionVol. 10(11)
Stony Brook University (US)
Interactive and Immersive Displays
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon