Deep Learning and visual search: Using raw eye movement data, convolutional neural networks generate target-location predictions in line with experimental manipulations

Eye tracking during visual search generates spatiotemporally rich but complex data. Traditional analysis methods typically employ metrics like the proportion of trials containing a saccade to a target (or distractor), dwell time on important stimuli, etc. However, such approaches leave out potential information contained in the raw eye data. Here, we asked if deep learning advancements might aid scientists in navigating this trade-off. A convolutional neural network (CNN) is a type of artificial neural network that can identify key features of input data, then use these features to sort (classify) unlabeled inputs into their appropriate groups. CNNs can learn from this classification process, using mistakes to help further define the portions of data that are most informative in determining what group an input belongs to. Although CNNs are commonly applied to images, they can generate predictions from other complex inputs like timeseries data. In a pre-existing dataset, participants searched for a color-defined target amongst 5 differently-colored distractors. We built a CNN that receives the raw x,y timeseries data and predicts which of the six locations contained the target on each trial. We trained the CNN on 2/3 of the data and validated on the rest. In short, the CNN performed well, predicting target location substantially above chance (67% vs. 17%). In our study, participants were pre-cued with reliable information about target color on half the trials (validity: 100%) and with unreliable information on the other half (validity: 50%). Prediction accuracies of two new CNNs, trained and validated on data from the two precue conditions separately, reflected this experimental intervention: greater classification accuracy in reliable (70%) than unreliable trials (63%). Bootstrapped error bars and subject-level null hypothesis testing confirmed the statistical reliability of this difference. These findings highlight the potential of CNNs as a novel analysis method.