TESTIFAI: Probabilistic Context-Aware Testing for Safe Deep Learning Models

Deep neural networks (DNNs) are integral to AI systems deployed in safety-critical domains like autonomous driving, healthcare, and cybersecurity. However, assessing their robustness in real-world scenarios remains a significant challenge. Traditional metrics like neuron coverage often fail to quantify the sensitivity of DNN models to input perturbations. We introduce TESTIFAI, a probabilistic framework to comprehensively evaluate and enhance DNN robustness across diverse, context-aware scenarios. Testifai comprises four key stages: (1) specification enables users to simulate real-world test scenarios in the form of robustness properties (e.g., sensitivity to image rotation), their configuration (e.g., range of rotation angles), and their statistical inter-dependencies; (2) mapping constructs a probabilistic coverage graph that captures the aforementioned dependencies, updating probabilities as testing progresses; (3) test-case generation systematically produces property-specific unit tests; and, finally, (4) execution calculates both local (i.e., property-specific) and global (i.e., system-wide, based on property dependencies) coverage metrics. Our initial experimental results demonstrate that TESTIFAI offers a rigorous and adaptable approach to DNN testing suitable for complex, safety-critical environments.