SleepLM: Modality Alignment of LLMs with EEG Signals

EEG signals are critical for various medical tasks, but current methods for EEG-based downstream tasks often lack a unified model capable of handling multiple tasks simultaneously. While large language models (LLMs) have demonstrated strong multi-task capabilities in the natural language processing (NLP) domain, their application to EEG signals remains underexplored due to their limited understanding of non-textual data. In this work, we address this gap by introducing a novel method that aligns EEG signals with LLMs. We begin by using VQVAE to discretize EEG signals into tokenized representations. These EEG tokens, along with textual data, are then used for autoregressive pretraining of the LLM, allowing it to learn both language and EEG signal patterns in parallel. Finally, the pretrained model is fine-tuned on specific downstream tasks, such as sleep stage classification and seizure detection, using a portion of the labeled data. Our experimental results show that this approach significantly improves performance on EEG classification tasks, outperforming traditional methods that do not leverage LLMs, and demonstrating the potential of LLMs for multi-modal learning with EEG signals.