AccessibleMelaka: A Computer Vision-Based Multi-Modal System for Enhancing Communication and Navigation for Disabled Individuals

Individuals with disabilities (hearing impaired, mute and color blind) face significant communication and navigation challenges in urban environments, especially in tourist environments such as Melaka, Malaysia. Current assistive technologies typically address single impairments without providing integrated solutions. This paper presents AccessibleMelaka, a comprehensive computer vision assistive system designed to enhance accessibility for individuals with disabilities in Melaka. The system integrates three modules: Sign Language-to-Speech (SL2S), Speech-to-Sign Language (S2SL) and Traffic Light Color Detection (TLCD). The system uses MediaPipe for real-time hand gesture detection, natural language processing for linguistic translation and YOLOv8 for object detection. Experimental evaluation demonstrated robust performance: SL2S achieved 97.20% accuracy for verb recognition, S2SL showed 95.98% accuracy for direct translations, and TLCD provided reliable real-time traffic signal detection with 0.92 precision and 0.89 recall. AccessibleMelaka effectively bridges communication gaps and enhances mobility for disabled individuals, contributing to a more accessible urban environment.