EEG-Driven Real World Image Captioning for Visually Impaired

This paper presents a novel proof-of-concept assistive system that links imagined speech with real-world image captioning, aiming to assist individuals with visual impairments. Our system detects and interprets the mental intent of a user by analyzing Electroencephalography (EEG) activity signals rather than physical approaches. It activates a camera embedded in the apparatus to capture the surrounding environment. The captured image is processed through a captioning system that uses Contrastive Language-Image Pre-training (CLIP), a state-of-the-art vision-language model, to generate a precise textual description. This text is then converted into speech using Google Text-to-Speech (gTTS). All of this is carried out with the objective of minimizing latency. Our trigger system achieves an accuracy of 63% using the 2D Spectrogram CNN, for faster processing, although other slower models such as Random Forest Classifier (63%) and Hist Gradient Boosted Classifier (70%) perform slightly better. Our system shows promising potential to revolutionize assistive technology through brain-computer interfacing, offering visually impaired individuals a new pathway to environmental awareness without physical interaction.