Post-hoc Uncertainty Quantification for Neurosymbolic Artificial Intelligence

The proliferation of deep neural network architectures into the space of mission-critical applications raises great risks given the lack of transparency behind model predictions and the paucity of reliable metrics to quantify model uncertainty. By combining certainty and competence scoring, derived from a recent post-hoc uncertainty quantification framework, and with the use of logic tensor networks (LTN), a form of neurosymbolic artificial intelligence, we examine and make improvements to the baseline xView2 neural network architecture developed for disaster damage assessment for aerial image data. This framework is able to identify gross incompetence on the part of the baseline model at damage assessment despite its superficially strong performance with unbalanced data, while also demonstrating the relative resilience of LTNs that incorporate the baseline model as a trainable feature. In particular, we find that the baseline model absent additional logical information collapses in accuracy to a uniform guess, with attendant collapses in F1 score, whereas the LTN improves across measures of accuracy, F1 score, and various measures of certainty, competence, and logical satisfaction, motivating further development of these tools in mission critical settings with pervasive structural uncertainties.