Breakpoint Resumption Migration Method of Two-Level Sharding for Single Table with Billion Data

With the arrival of the big data era, tens of billions of single-table big data generation, the increasing data scale shows how to efficiently and quickly big data migration has become a key challenge. Current traditional big data migration methods have the problem of slow migration speed, and the risk of abnormal interruptions. This paper proposes a method for breakpoint resumption migration of secondary sharding for single-table data at the billion-scale level. Dividing primary shards into secondary shards, and employing a dynamic load balancing strategy to migrate to multiple databases enhance the independence and controllability of data migration, consequently reducing overall migration time. Additionally, in case of any anomalies during the migration process, simply rolling back and re-migrating the affected data shards ensure minimal impact on other shards or the overall migration process. To evaluate the efficacy of this method, our study conducted experiments on migrating Oracle database to other databases. The findings confirmed that the proposed approach supports migration to diverse databases, and as the number of nodes increases, migration efficiency grows exponentially. Additionally, comparative experiments were conducted between the data migration tool developed by this method and both DataX and the DaMeng Transfer Service (DTS), with equivalent data volumes. The experimental results indicate that this method performs comparably well to the leading DTS in terms of migration efficiency. However, this method’s advantage lies in its support for distributed deployment, resulting in the highest overall migration efficiency. Furthermore, unlike other migration tools, it also supports migration to multiple database platforms.