C²HI: Towards a Command and Control Hierarchical Interface for Human-Robot Teams in Dynamic Social Environments

Human-robot teams (HRTs) in dynamic domains like defense and search & rescue require hierarchical interaction: the ability to consider a human’s role within a hierarchy and environmental context. However, most robot perception systems lack this capability. We present the Command & Control Hierarchical Interface (C<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>HI), a specialized perception framework using Dynamic Bayesian Networks (DBNs) and multiagent assignment to fuse asynchronous, multimodal sensor data. C<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>HI concurrently estimates scene context, human positions, roles, and commands to enable hierarchical awareness, a critical precursor to more advanced teaming behaviors. Using an audiovisual dataset recorded in dynamic social scenes, we demonstrate the framework’s capabilities, notably achieving high visual role recognition accuracy (> 90% after 5 observations using CLIP foundation model) comparable to using a fiducial marker for role recognition. While fiducial commanding was effective (69% accuracy), verbal (31%) and gestural (24%) command recognition proved challenging in these open environments, indicating modality-specific limitations. Our open-source dataset and code encourage further development of hierarchical command and control for HRTs.