Reliability and generalization of EEG-based deep learning for consumer-preference prediction: a systematic review and evidence synthesis

AbstractA growing body of work uses deep learning to classify EEG recordings into categories corresponding to different consumer preferences. A key issue is its generalizability to new consumers who were not included in the development of the model. To investigate this question, we conducted a systematic review of studies using EEG-based consumer preference classification by deep learning. Our searches identified 7,480 records. Of those, 39 reports were included in the review. The 39 reports comprised 24 underlying dataset/study records and contained a total of 130 model results. Twenty-seven of the reports had a high concern for data leakage. Only nine of the reports evaluated the performance of their models on new consumers. None of the studies established robustness across sessions. Only one study investigated transfer to a new dataset recorded with a different EEG device. However, generalization to new consumers is not yet adequately supported by the evidence. In repeated-measures designs, partitioning trials or time windows into training and test sets instead of consumers could lead to overoptimistic results, because data from the same consumer may be included in both sets. Future reports on EEG-based preference classification should therefore not only report results on the accuracy of their models but also provide independent evidence of performance on new consumers, in different sessions, with different devices, and on different datasets. The review protocol was finalized before searching, but it was not publicly registered.