DMML: A Machine-learning Performance Model for Data Migration

Data migration at scale can be a daunting task. It may require significant resources and time, which must be taken from value-adding activities of an enterprise. Besides errors may occur, which can jeopardize the integrity of the data and waste resources. Accurately estimating data migration time and resource performance is critical for optimizing time, cost, and risk in large-scale data transfers. In this paper, we propose the use of machine learning to create performance models for data migration. We utilize DMBench, a benchmarking and load testing tool specifically tailored for data migrations, to generate data, simulating various data migration scenarios with different data sizes, vCPUs, RAM size, and data compression types. We experimented with multiple ML algorithms and showed the effect of hyperparameter tuning in the model's accuracy. Our results show that the XGBoost is the most accurate and consistent across the different scenarios. We demonstrate the model building process and its evaluation on an industrial case study.