Intuitive, phenomenological approach to test methods for approval purposes of trained models using the example of predicting loudness classes from EEG-data

Abstract Machine learning (ML) models will soon be found in devices of critical structures or even in high-risk applications. Arranging the Artificial Intelligence Act of European Union, requirements and a methodology for an approval of measurement devices by a regulatory authority or a state institution need to be developed. In this study, using the example of loudness classification from electroencephalography measurement data, phenomenological methods were developed for the evaluation of ML models in the context of an approval process. A schedule of six sequential steps is proposed that can be used as a flowchart in such a process. After identifying the relevant features (step 1) performance measures and benchmarks were defined (step 2). In the following performance evaluation (step 3) the influence of the input data on the model’s output was determined. The range and characteristics of the input data at which the device operates correctly were identified and quantitative descriptors were determined for the full range of operating conditions specified in standards or regulations for approval (step 4). Perturbation-based techniques were developed and adapted to define corner cases and to create new test data that identify the areas of the input data where performance falls below a certain threshold (step 5). The newly found test data were finally applied in step 6, and it could be shown that the data can reveal weak points in the models. All processes can and must be used without knowledge of the models, as an independent test of a finished device is always carried out during a type approval. Thus, this study contributes to the still very nascent field of type approval development for measuring devices, software and methods containing machine or deep learning.