AI Maintenance Techniques by Detecting Performance Degradation in Domain Shift Using Model Ensembles

We propose the method to automatically detect performance degradation due to domain shifts, such as changes in location and weather, for object detection AI. The algorithm is based on the conventional method of estimating uncertainty using the diversity from a deep ensemble. However, by introducing a “background class”, models that do not output bounding boxes (i.e., models with false negatives) which were previously ignored are also included in the uncertainty definition. The correlation between this new uncertainty and model performance was evaluated using a custom-created gas station dataset, and showed a strong negative correlation (−0.96 or less) compared to the conventional method. This results indicats that the proposed method considering the false negative model can accurately detect performance degradation.