Evaluating haptic experience using EEG and deep learning across multiple modalities: linking stimulus and self-reports

Conventionally, evaluations of haptic interfaces have relied on self-reported assessments, which offer limited objectivity and can disrupt the user experience, making it challenging to design interfaces that dynamically adapt to users' cognitive state in real time. To overcome these limitations, cognitive haptic interfaces leverage neurophysiological measures such as EEG and deep learning to directly capture the brain's responses to haptic stimulation. A key challenge is how to label these neural responses: do we ground models in objectively controlled Physical Stimulation (PS) parameters, or in participants' Self-Reported (SR) perceptions? The goal of this work is not to demonstrate that EEG can reproduce subjective reports, but rather to systematically examine how neural responses relate to these two aspects of haptic experience by training deep learning models under both PS and SR labeling schemes. Here, we investigate how PS- versus SR-based labeling impacts model performance across four modalities: (i) delayed force-feedback (DFF), (ii) fingertip vibration feedback (FVF), (iii) upper-body vibration feedback (UVF), and (iv) fingertip thermal feedback (FTF). We evaluate three deep learning benchmarked architectures: ATCNet, EEG Inception, and EEG Conformer on EEG data labeled according to both approaches. Across all modalities, PS-labeled models yield more stable and higher performance than SR-labeled models in a group-level leave-one-subject-out (LOSO) setting, with the largest gains at near-perceptual-threshold levels (e.g., mild thermal changes, moderate vibration intensities, borderline delay settings) where SR labels are most variable across individuals. Rather than aiming to replace self-reports, these results reveal when EEG-based models align more closely with the physical stimulation than with participants' reports and support using PS-trained decoders as a structured first-stage representation that can later be adapted with user-specific SR information.