Mitigating Data Hotspots in Distributed Cloud Storage Architecture

Cloud storage is a vital component of cloud architecture, often utilizing distributed key-value stores like Amazon S3 and Google Cloud Storage for managing da-ta and metadata. These systems distribute data across nodes using key range or consistent hashing, but they face challenges such as load imbalance and limited parallelism due to uneven data distribution and varying node performance. Cur-rent implementations, such as MongoDB, address these imbalances by migrating data between nodes but often neglect the characteristics of the underlying data structures, leading to increased overhead from costly delete and insert operations. To address these issues, this design leverages the properties of the LSM tree, a commonly used storage engine, to optimize data migration. The approach intro-duces hot zone prediction using nonlinear regression to accurately identify data hotspots based on key characteristics, insertion time, and TTL. A storage engine-aware migration system is developed to migrate grouped SSTable files rather than individual key-value pairs, significantly reducing migration overhead. Additionally, the data migration I/O process is offloaded using the NVMe-oF protocol, minimizing CPU involvement and preserving node performance. Implemented on mongo-rocks, this solution improves load balancing by directly moving SSTable files across nodes, enhancing efficiency and reducing performance degradation in distributed key-value stores.