Safe AI for coral reefs: Benchmarking out-of-distribution detection algorithms for coral reef image surveys

Although deep learning has demonstrated significant advances in qualitative domains, deep learning algorithms remain poor at quantifying the uncertainty of their predictions. This is especially true when applied in domains where there is data shift outside of which an algorithm has been trained. This has major implications for the use of deep learning in accurately estimating change in environmental monitoring applications. In the case of image classification for coral reef habitats, time series imagery is rarely consistent due to changing environmental conditions, differing sensors and inconsistencies in data capture, which means traditional machine learning metrics simply do not work when applied to new out of distribution datasets. 1. For this reason, we benchmark the latest state-of-the-art OOD (Out Of Distribution) detection algorithms on publicly available coral reef image datasets, and evaluate histogram intersection of outlier scores as an indicator for human intervention. 2. We show with a comparative analysis that the performance of OOD detection algorithms is variable, and highly dependent on in-distribution and out-of-distribution data composition. We show that KNN (K-Nearest Neighbour) distance was the most consistent across datasets, followed by Virtual-logit matching (ViM). 3. This research shows a compelling example of how a handful of openly available algorithms can easily be used as a complimentary indicator alongside confidence (Softmax probability), in turn providing more efficient and safe deployment of deep learning for rapid reporting of coral reef habitats. • Deep learning algorithms are poor at quantifying the uncertainty of their predictions, especially on data with which a model has not been trained. • There exists a handful of off-the-shelf algorithms that can be utilized to detect out-of-distribution data and indicate when a model is unable to make an accurate prediction. • These algorithms can be adopted immediately by practitioners utilizing deep learning for image classification in coral reef image surveys and any ecological workflow that uses images for environmental reporting.