Bio-Inspired Gaze and Neural Command Fusion for Assistive Smartphone Interaction

This paper presents an assistive smartphone interaction system that combines mobile gaze tracking with EEG-based BCI commands. The Android application estimates the user’s gaze with the front camera, MediaPipe facial landmarks, a TinyTrackerS TFLite model, temporal smoothing, and a calibrated mapping from model output to screen coordinates. The gaze point is used to locate the intended screen area, while the BCI layer uses Emotiv Cortex commands for click, scroll, and back actions. A FastAPI and MongoDB backend manages profiles, calibration data, validation reports, runtime data, and WebSocket control events. Android Accessibility is used to execute the selected actions, with raw tap fallback when needed. The system was tested with 36 student volunteers during a short Patient Assist task. In the evaluation, 33 out of 36 gaze mappings were promoted to the active profile. The average static mean error was 390.02 px, and the average static p95 error was 789.09 px. BCI command success was 89.58% for click, 72.22% for scroll, and 77.78% for back. The Android layer acknowledged 228 out of 234 accepted control events. The average usability score was 4.21 out of 5.