Human and algorithmic visual attention in driving tasks

Amid the heated debate on whether artificial intelligence possesses a human-like capacity for understanding, the compatibility and interaction between human and algorithmic visual attention remain unclear. Here, we address this issue through the lens of spatial and feature-based attention. Using autonomous driving as an epitome of safety-critical domains, we show that human attention in driving tasks can be divided into three phases, each characterized by spatial, feature-based, and mixed visual attention. Comparisons between each phase of human attention and algorithmic attention revealed a complex landscape of human-AI resemblance. For specialized detection and planning algorithms, incorporating semantic-rich, feature-based human attention markedly enhanced performance, suggesting these models lack human-like semantic visual understanding. In contrast, for large-scale Vision-Language Models, the effect was task-dependent, suggesting that while foundation models have bridged the “reasoning gap” through massive pre-training, a “grounding gap” persists in fine-grained visual tasks. Crucially, our findings demonstrate that incorporating human semantic attention offers an effective and economic pathway to compensating for these gaps, enhancing model understanding in safety-critical and grounding-heavy tasks without the need for massive scale.