Recovering Missed Detections in an Elevator Button Segmentation Task

One obstacle that mobile service robots face is operating elevators. Reading elevator control panel buttons involves both an instance segmentation of buttons and labels and associating buttons with their respective metal labels in the elevator. Segmentation algorithms, however, can miss detections. This paper presents a segmentation model specifically designed to solve the problem of missed detections. This can be used to recover detections that the initial model misses. This work presents: 1) a new elevator button dataset containing both 108 images sampled from the internet and 292 images imaged from 24 buildings from the University of Texas at Austin campus and the surrounding neighborhood, along with their segmentation boundaries and associated labels; 2) a vision pipeline based on Mask-RCNN for solving the initial image segmentation and labeling task; and 3) a novel method for identifying missed detections, using a Mask-RCNN network trained on expected button locations. Results show that the missed detections model, specifically developed to recover buttons and labels that were missed by the initial pass, is accurate on up to 99.33% of its predicted missed features on a synthetic missed-detection dataset and 97.14% of its predictions for features missed on a non-synthetic dataset. In the case of the average accuracy of successful button and label detections of a specifically-trained "weak" initial detector at a standard IoU threshold of 0.5, the missed detection model improves the detector’s success rate from 80.38% on the button recognition task with the initial segmentation model only to an average accuracy of 90.7% with the missed detections model enabled. The overall accuracy of the best-performing pipeline implementing the missed detections model is 91.73% and 98.27% on our Internet subset and Campus subset of our dataset, respectively.