Generalising AI model performance for Non Destructive Testing in railway systems

Maintaining railway systems is an essential task during operation of such public infrastructure, but the methods have remained unchanged in Germany for decades. Advancements in ML modeling, Simulation Techniques, and Data Management enable possibilities to accelerate the shift to more efficiently digitized evaluation processes. This research aims to analyze the capabilities of generalization of a previously identified Deep Learning model, trained on real-field ultrasonic data from regular inspection runs, and augmented by simulated defects. As this cannot be done by the usually applied summary statistics, it is necessary to analyze the model performance following the questions of how well the model learns data pattern explicitly, can interpolate between the trained parameters, and whether it did not overfit on specific simulation pattern. This is done by applying selective sampling strategies and an analysis of interesting results. The findings indicate a good fit for explicit patterns with an AUC of 0.964, while interpolation between parameters is successful only in specific use cases. Finally, we conclude that the model likely learns background patterns from simulations and may not necessarily apply simulated defects to real-world scenarios.