Electrophysiological evidence for differential semantic processing of words and objects presented in augmented reality

Head-worn augmented reality (AR) offers exciting possibilities to help users learn new information. By blending digital content with the learner's real-world environment, AR can create engaging and enjoyable experiences that may improve knowledge retention. Electroencephalography (EEG) allows for discreet, continuous monitoring of brain activity at the scalp. This study examined whether the N400 event-related potential (ERP), a brain response linked to semantic processing, could be incorporated into a system combining AR and EEG. While the N400 is reliably elicited by a mismatch in meaning between two sequentially presented stimuli, there are two key outstanding questions. First, how do 3D objects presented in AR impact semantic processing as measured by the N400? Second, is there a reliable N400 to mismatches between an object and its name, in addition to mismatches in meaning? Twenty-four young adults viewed sequential pairs of stimuli through an AR headset while EEG was recorded. We manipulated whether the first stimulus was a 3D object or written word and whether the second stimulus matched or mismatched the first in terms of meaning or name. Participants' reaction times were slower for mismatching pairs when compared to matching pairs in all conditions. Averaged ERP and single-trial classification analyses showed robust differences in brain responses. Additionally, participants were more accurate and exhibited faster behavioral and brain responses for naming compared to meaning judgments. These results suggest the N400 could be used in a combined AR-EEG system to provide feedback on semantic understanding, potentially opening exciting new avenues for enhancing learning. • Augmented reality (AR) offers exciting possibilities to help users learn new information. • Electroencephalography (EEG) is a noninvasive method for monitoring brain activity. • We combined AR and EEG to examine an EEG signal linked to meaning, known as the N400. • We demonstrated robust N400 responses in all conditions. • Results are a first step towards an AR-EEG system for semantic knowledge detection.