Integrating Human Feedback in VR – A Human-in-the-Loop Approach to Real-Time Gesture Recognition

Studies in virtual reality (VR) often suffer from missing participant feedback due to limited sensor data available on VR devices. However, some experiments require direct feedback to be evaluated in the VR scene immediately, which tends to be challenging. Our setup utilizes a coordinated setup of hardware and software components to provide a feedback loop from participants' gestures into the VR scene so entities can react to people's actions while simultaneously collecting data from all connected systems. This demo showcases a study setup including (a) a central data broker with low- and high-frequency data storage, (b) a gesture recognition and publishing system, (c) a VR system to collect entity movement data and voice transcriptions, and (d) a command-line interface for managing the system. This should facilitate study execution by limiting waiting times for participants and improving the overall data collection process.