Evaluating Methods for Scenario Reasoning using Bayesian Networks in Exhaustive and Non-Exhaustive Settings

Tunnel vision and confirmation bias can lead to miscarriages of justice. A way to avoid tunnel vision is to consider your evidence in light of more than one scenario. Alternative scenarios allow us to consider how probable each scenario is, compared to the other considered scenarios. Bayesian Networks have been proposed as a formal method for reasoning about the probability of scenarios. Specifically, alternative scenarios were modelled using Bayesian networks with a constraint node, which ensures mutual exclusivity. However, the performance of these methods in situations where not all possible alternative scenarios are modeled, the non-exhaustive setting, has not been investigated. Since it is impossible to explicitly cover everything that could possibly have happened in a model, it is important to know how these methods handle non-exhaustiveness. We evaluate four methods using an agent-based model that simulates an environment in which a crime could occur. Taking this as the ground truth, we compare different Bayesian network modeling methods on five aspects related to the quality of the representation of the ground truth as well as computational performance. We find that some methods result in disparities between the ground truth and the predicted posterior probabilities for the scenarios in a non-exhaustive setting. In an exhaustive setting, the proposed methods perform well. The construction approach that models scenarios in terms of conjunctions of events performs well in both settings.