Multi-faceted sensory substitution using wearable technology for curb alerting: a pilot investigation with persons with blindness and low vision

Curbs separate the edge of raised sidewalks from the street and are crucial to locate in urban environments as they help delineate safe pedestrian zones from dangerous vehicular lanes. However, the curbs themselves are also significant navigation hazards, particularly for people who are blind or have low vision (pBLV). The challenges faced by pBLV in detecting and properly orienting themselves for these abrupt elevation changes can lead to falls and serious injuries. Despite recent advancements in assistive technologies, the detection and early warning of curbs remains a largely unsolved challenge. This paper aims to tackle this gap by introducing a novel, multi-faceted sensory substitution approach hosted on a smart wearable; the platform leverages an RGB camera and an embedded system to capture and segment curbs in real time and provide early warning and orientation information. The system utilizes a YOLOv8 segmentation model which has been trained on our custom curb dataset to interpret camera input. The system output consists of adaptive auditory beeps, abstract sonifications, and speech, which convey curb distance and orientation. Through human-subjects experimentation, we demonstrate the effectiveness of the system as compared to the white cane. Results show that our system can provide advanced warning through a larger safety window than the cane, while offering nearly identical curb orientation information. Future enhancements will focus on expanding our curb segmentation dataset, improving distance estimations through advanced 3D sensors and AI-models, refining system calibration and stability, and developing user-centric sonification methods to cater for a diverse range of visual impairments.