GQA Network: Research On Pilot Motor Imagery Decoding Based On Multimodal Data Fusion of EEG and Visual Fixation

The classification of exercise imagination based on pilot physiological and behavioral information is of great significance for aviation human–computer interaction and pilot training. This study proposes a multimodal fusion framework for grouped query attention (GQA), which enhances the detection ability of pilots’ motor imagination by fusing EEG and visual gaze information. Method: First, arrange flight trainees to conduct a five-sided flight experiment task in the simulator, synchronously collecting a 32-channel EEG, visual gaze, flight control data, and flight record data. Then, organize flight records and operational data, analyze the characteristics of the pilot’s movement of the control stick for time window segmentation, and combine flight data to determine the pilot’s motion imagination category under the window based on climb, descent, left turn, and right turn. Extract EEG and gaze data corresponding to the time window to form a dataset. Construct a neural network model based on GQA, which takes in EEG and visual gaze time-series data and outputs a classification structure for pilot motor imagery. Utilize the inference efficiency and performance balance advantages of grouped query attention (GQA) to improve detection efficiency and meet real-time monitoring needs. Result: The average classification accuracy of this model can reach 91%, which is better than traditional models such as LSTM, and runs faster than multi-head attention (MHA). Based on the framework of GQA, it can effectively explore the effective value of physiological behavioral data and provide a reliable technical solution for the objective detection of pilot motion imagination.