Effects of Incongruencies Across the Reality-Virtuality Continuum

This dissertation examines the perceptual and cognitive effects of incongruencies in eXtended Reality (XR) experiences along the Reality-Virtuality (RV) continuum, with a particular focus on Virtual Reality (VR) and Video See-Through (VST) Augmented Reality (AR). VST AR integrates video images from front-facing cameras on the Head-Mounted Display (HMD) with virtual content. XR HMDs, that are capable of VST AR, oftentimes also include a VR mode. While VR has been extensively studied, VST AR remains underexplored despite rapid advances in camera resolution and rendering techniques. The blending of virtual and real-world elements in VST AR frequently gives rise to perceptual mismatches, such as conflicting depth cues, misaligned virtual objects, and latency discrepancies, that challenge established XR frameworks and may adversely affect user experience. This dissertation, incorporating five key publications and five empirical experiments, investigates effects of incongruencies in VR and VST AR, by examining both subjective reports and objective behavioral measures. While users may not always consciously detect these mismatches, the empirical findings of this dissertation reveal their significant impact on depth perception, spatial judgments, and performance. A central focus of this work is the application and refinement of the Congruence and Plausibility (CaP) model, which describes how incongruencies operate at different processing levels — from low-level sensory distortions to higher-order cognitive inconsistencies. The results indicate that AR-inherent perceptual incongruencies influence the experience at a subconscious level, challenging existing theoretical frameworks that primarily focused on VR experiences that are visually coherent. To further support this understanding, the dissertation introduces a methodological framework for analyzing and predicting the effects of incongruencies, contributing to the development of coherent and immersive XR applications. The conducted research affirms both the complexity and promise of VST AR technologies. By disclosing how subconscious factors interact with users’ conscious perceptions, this dissertation enriches theoretical understanding and provides strategies for advancing XR research.