Forecasting Changes in the Data Characteristics of AI Models

We introduce a framework designed to forecast changes in the characteristics of data fed into machine learning models' using chest X-ray (CXR) imaging as a case study. Our framework investigates the prediction of three types of changes in data patterns: seasonal, gradual, and incremental. We employed three forecasting methods to predict these changes: Long Short-Term Memory (LSTM), Seasonal Auto-Regressive Integrated Moving Average (SARIMA), and Holt-Winters. Our results indicate that while LSTM achieved the best overall performance in terms of mean absolute percentage error (Seasonal<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MAPE</inf> = [10.27], <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Incremental_{MAPE}$</tex> = [0.87], <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{Gradual}_{MAPE} =[7.35])$</tex>, it struggles to accurately capture seasonality in future predictions, a task where the Holt-Winters model performs more effectively. Our findings demonstrate the feasibility of predicting changes in data characteristics provided to machine learning models. Such early prediction is needed as data drift, due to the dynamic nature of real-world environments, may affect model performance during deployment. By anticipating these changes, our framework can enable proactive measures for maintaining the reliability and safety of deployed AI models.