Towards Agile and Judicious Metadata Load Balancing for Ceph File System via Matrix-based Modeling

To scale out the massive metadata access, the Ceph distributed file system (CephFS) adopts a dynamic subtree partitioning method, splitting the hierarchical namespace and distributing subtrees across multiple metadata servers. However, this method suffers from a severe imbalance problem that may result in poor performance due to its inaccurate imbalance prediction, ignorance of workload characteristics, and unnecessary/invalid migration activities. To eliminate these inefficiencies, we propose Lunule, a novel CephFS metadata load balancer, which employs an imbalance factor model for accurately determining when to trigger re-balance and tolerate unharmful imbalanced situations. Lunule further adopts a workload-aware migration planner to appropriately select subtree migration candidates. Finally, we extend Lunule to Lunule + , which models metadata accesses into matrices, and employs matrix-based formulas for more accurate load prediction and re-balance decision. Compared to baselines, Lunule achieves better load balance, increases the metadata throughput by up to 315.8%, and shortens the tail job completion time by up to 64.6% for five real-world workloads and their mixture, respectively. Besides, Lunule is capable of handling the metadata cluster expansion and the workload growth, and scales linearly on a 16-node cluster. Compared to Lunule, Lunule + achieves up to 64.96% better metadata load balance, and 13.53-86.09% higher throughput.