Fine-Grained Dynamic Hardware Resource Usage Measurement for Distributed Key-Value Stores

In the era of big data, distributed key-value stores are widely used in various fields. These systems efficiently store and manage massive amounts of data by distributing data across different. It also splits data into multiple fine-grained shards for management purposes. However, the minimum unit of hardware resource usage perception is the node because it is difficult to measure the hardware resource usage of the shard since it is a logical structure, and the corresponding data is scattered in different locations in the nodes. This inconsistency in granularity leads to challenging hardware resource usage management while the unit of management is shard. To solve this problem, we propose a new hardware resource usage measurement framework FDHM, which integrates a measurement model based on the shard and node attributes and a dynamic attribute update mechanism to achieve fine-grained and dynamic measurement. We implemented a lightweight integration of FDHM on TiKV and tested it. Experimental results show that FDHM significantly improves the accuracy of hardware resource usage measurement compared with existing methods. And we also verified its benefits in resource management through a case study.