Detecting attended auditory events using ear-EEG:new approaches and methods for auditory attention decoding

Hearing is fundamental to how humans interact with the world. Without hearing, we are significantly limited in our communication and interaction with the world. Hearing impairment may lead to depression and feelings of isolation. Unfortunately, a large proportion of the population has been diagnosed with hearing loss, which is a partial or total inability to hear. According to the World Health Organization, in 2021, 432 million adults and 34 million children are suffering from hearing loss. By 2050, it is estimated that over 10\% of the world’s population will have hearing loss. A common solution for individuals to deal with hearing loss is using a hearing aid which is designed to compensate for the lost components of the sound, making it audible to a person with hearing loss.<br/><br/>Despite tremendous advancements in hearing aid technology over the past decades, challenges remain. Particularly, in environments with multiple active speakers, current hearing aids offer limited assistance, as they lack information about which speaker to target. If the hearing aid could identify the speech source of interest, i.e., decoding auditory attention, it could enhance the listening experience by suppressing irrelevant speech and amplifying the desired signal. Auditory attention decoding (AAD) emerges as a promising technology in this context and is becoming a trending research area for the next generation of hearing aids. In this thesis, I present a novel approach to auditory attention detection: using the endogenous processing of auditory events. This will combine state-of-the-art machine learning methods with deep analysis of auditory event-related cognitive features of the scalp- and ear-EEG signal, using recordings from normal-hearing subjects.<br/><br/>In the first part of this thesis, I investigate the high-level cognitive responses to natural speech events that occur during auditory attention. Specifically, I present four experimental paradigms with increasing degrees of realism: a word category oddball paradigm, an oddball paradigm with competing speakers, and competing speech streams with and without specific targets. EEG data has been recorded from 24 participants using 32 scalp electrodes and 12 in-ear electrodes (ear-EEG) to conduct a study of cognitive components related to speech events. I find that cognitive processing of natural speech events is observable at parietal electrode sites, peaking around 620 ms, likely corresponding to the P3b component. Additionally, in multi-talker paradigms, this component remains detectable in the attended speech stream, and its amplitude is affected by the degree of attention engaged by the brain to the speech events. These findings suggest that this component is a potential feature for decoding auditory attention. Furthermore, by using the spatial filtering method, the component can be extracted from ear-EEG signals.<br/><br/>In the subsequent part of the thesis, I develop a deep learning model to detect this cognitive component within single EEG epochs associated with speech events. This addresses the single-word AAD problem, determining whether a single word is attended or not. The results demonstrate that the proposed model can exploit cognitive-related spatiotemporal EEG features, achieving promising results on of the most realistic competing paradigm for unseen subjects. Importantly, this result suggests a direction to solve the AAD problem by using extended deep learning models for a series of cognitive responses to auditory events in a speech stream.<br/><br/>Next, we focus on enhancing current state-of-the-art AAD models. Traditional AAD methods typically include two separate stages: reconstructing the envelope or a third-party representation of the attended speech and determining the attention by comparing the similarity between the reconstruction and the actual representation of all candidates. I introduce AADNet, an end-to-end deep learning model, which combines these two stages into a direct approach to address the AAD problem. The proposed AADNet outperforms the competing methods including linear stimulus reconstruction, canonical correlation analysis, and non-linear stimulus reconstruction for both subject-specific and subject-independent models on two different datasets. The results highlight improved generalization to data recorded on unseen subjects and demonstrate the promise of using a deep learning approach to enhance AAD performance.<br/><br/>Finally, I apply the developed AADNet and propose a new sparse representation of speech signals to adopt the cognitive response-based approach to address the AAD problem and validate the performance on both scalp and ear-EEG data. I find that the proposed sparse representation correlates with the neural responses and this correlation can be used to decode the auditory attention. Although the proposed method currently performs modestly compared to envelope-based approaches and is limited to only scalp EEG, these results demonstrate the potential of utilizing cognitive response features rather than the stimulus-driven response to address the AAD problem.