Synthetic Versus Real Images for Indoor Building Defect Classification: Evaluating the Synthetic-to-Real Performance Gap

Building defect datasets are slow to assemble and expensive to label, and indoors there is the added problem of photographing people’s homes. Text-to-image models look like an easy way round this, but it has not been shown that a classifier trained on generated pictures will recognise defects in real ones. We repeated an earlier experiment on three moisture-related indoor defects (damp, mould and paint deterioration) in two arms. The same ImageNet pre-trained VGG-16 network, set up as in the earlier study apart from the number of epochs, was trained once on real, manually collected and labelled images and once on AI-generated images (Stable Diffusion XL 1.0). Tested on images from its own domain, the synthetic-trained model scored 91.7% accuracy (macro-F1 0.918, n = 303) and the real-trained model 83.2% (macro-F1 0.832, n = 559). That comparison is misleading: the test sets differ, and the synthetic test images share a generator with the synthetic training images. On a common set of 150 real photographs (50 per class), the real-trained model reached 82.0%, while the synthetic-trained model reached only 45.3% (95% confidence intervals 75.1–87.3% and 37.6–53.3%). Recall for damp fell to 8%; mould held at 92%. In the configuration we tested, AI-generated images did not work as a stand-alone replacement for real annotated images, and the apparent advantage over manual labelling disappeared on real photographs. We suggest treating synthetic images as a supplement to real data, and testing any synthetic-trained model on real images.