Applying Software Engineering Black-Box Methods for Testing Machine Learning Models

This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.