BEI-DETR: A Multimodal Remote Sensing Object Detection via Integrating EEG and Eye Movement

Recently, the DEtection TRansformer (DETR) and its variants have achieved success in object detection, but they still require manual intervention in complex scenes. Object detection methods cannot entirely replace human capabilities. Leveraging human perception to assist remote sensing object detection offers a new technical approach. In the process of manual verification, the most prominent signals reflecting human identification of specific targets are electroencephalogram (EEG) and eye movement. To our best knowledge, there is currently no publicly available dataset for this task. Therefore, we collect two types of human perception information while observing remote sensing images, forming a trimodal dataset including brain signals, eye movements, and remote sensing images. Upon this, we propose a multimodal object detection method, termed as BEI-DETR (Brain-Eye-Image DETR). Specifically, we decouple the classification and localization tasks in the object detection, using EEG for classification and eye movement information for localization. Subsequently, we propose a feature matching fusion module (FMFM) to fuse multimodal features and then put them into the decoder and prediction heads to obtain the final detection results. Comparative and ablation experiments demonstrate the satisfactory performance of BEI-DETR, indicating that our model can effectively learn human perceptual information for the remote sensing object detection task. The trimodal dataset and source code are publicly available at https://github.com/Hickey-Curry/BEI-DETR and https://github.com/Hickey-Curry/BEIDataset.