Investigating the Applicability of Image Generation Models to Weakness Detection Tasks

Sufficient testing against real-world ML model is mandatory because the safety violation of that behavior causes a huge social impact. Existing methods detect weaknesses in ML models using labeled features associated with data based on the concept of combinatorial test modeling. However, sufficient testing is difficult due to rare features that are difficult to collect from the real world, especially for image data. In this paper, we investigate an approach using an image generation model to solve this problem, and answer the following research question: To what extent can the generated data detect weaknesses by factor-level compared to the test data? Our experiments showed that $74 \%$ of the weaknesses detected by the generated data were detected by the test data as well. Our further investigations showed that the feasibility of generating data in a way that transforms factor levels is influenced by the generative model used for the transformation and the type of transformation.