Synthetic Data for Assessing Robustness of Deep Learning Models to Distributional Shifts

Ensuring robust performance of deep learning models across diverse target environments is a critical aspect of AI model development. However, acquiring a large variety and volume of evaluation data can be prohibitively expensive. To address this challenge, we propose leveraging synthetic data generated by generative models trained on existing data and informed by prior knowledge of deployment scenarios. By simulating multiple test distributions corresponding to distributional shifts, we rigorously evaluate model robustness. We illustrate our approach using the problem of multi-structure segmentation in echocardiographs, considering different racial demographics as test distributions. We observe that utilizing synthetic data can indeed evaluate models rigorously without additional data acquisition cost.