An intuitive and semi-automated transfer function design for interactive region of interest-based direct volume rendering in mixed reality head mounted devices

Direct volume rendering (DVR) enables the visual exploration of regions of interest (ROI) in a volume in an interactive manner. Mixed reality head mounted devices (MR-HMDs) have the potential to expand the utility of DVR. MR-HMDs enable not only the high-quality and stereoscopic DVR but also its in-situ visualizations of complementary data to assist in the real-world tasks of users. The state-of-the-art MR-HMD works, however, mainly focus on investigating the feasibility and effectiveness of DVR to target MR applications. The interactive ROI-based exploration of DVR has been severely limited in prior research, and user optimization is still confined to the primitive image manipulation of pre-configured ROIs (e.g., view rotation). In this work, we propose an interactive and semi-automated exploration approach for ROI-based DVRs in MR-HMDs by introducing a new MR-HMD-based rendering parameters (i.e., transfer functions (TFs)) optimization and utilizing the built-in biometric interfaces. Using eye-gazing and hand gestures, users can select ROIs within an initial DVR visually and interactively. Our MR-TF approach (named hereafter) then automatically optimizes the corresponding TF toward ensuring the visibility of ROIs. Additionally, our MR-TF approach addresses a critical rendering issue in MR-HMDs, i.e., the color perception of ROIs should be consistent regardless of the variations of intermixing physical backgrounds. Our MR-TF approach quantifies the color perception and integrates to the TF optimization for enhanced ROIs. We achieve on-device implementation of our MR-TF approach on the commercial MR-HMD with low computing capability, Microsoft HoloLens 2. Our evaluations with different types of volumes demonstrate the capabilities and practicality of our MR-TF approach.