From Neural Tokens to Language: Dual-View EEG Semantic Bridging for Brain-to-Text Generation
Visually evoked EEG-to-text generation aims to produce natural-language descriptions from neural responses elicited by visual stimuli. Existing approaches typically compress EEG signals into a single global representation and directly project it into the hidden space of a large language model (LLM), which may limit the preservation of complementary visual semantics under noisy and subject-dependent EEG signals. We propose a dual-view semantic token bridging framework that exploits raw images and sketch images as complementary visual supervision to learn raw-oriented and sketch-oriented EEG semantic token streams. To interface these structured neural representations with an LLM, we introduce a branch-specific semantic query bridge that aggregates each token stream into compact semantic slots and converts them into LLM-compatible soft prompts. A progressive adaptation strategy further transitions the learned visual-semantic interface from image-supported training to image-free EEG-conditioned generation. Experiments demonstrate strong subject-wise EEG classification performance and competitive EEG-to-text generation across multiple LLM backbones. Ablation, branch-level representation, and EEG-label disentanglement analyses further show that the two EEG streams learn complementary view-oriented representations and that EEG-derived prompts provide sample-specific information beyond category labels.
Authors
- Xiaozhao Fang (ORCID: https://orcid.org/0000-0001-8440-1765)
- Weijun Sun (ORCID: https://orcid.org/0000-0002-2342-4434)
- ZhuoShi Tang
- Jie Wen
- Jun Dai
- YaoYang Liu
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.48550/arxiv.2611.24197
- Primary Topic
- EEG and Brain-Computer Interfaces
- Type
- preprint