Data Redundancy Optimization in HDFS via Machine Learning Clustering Techniques

HDFS is the Hadoop Distributed File System, a critical part of big data storage, but processes suffer from lots of inefficiency due to duplicated data. In this paper, we propose to use machine learning as the core foundation of clustering algorithms such as K-means, DBSCAN, and hierarchical clustering to optimize data redundancy in HDFS. This entire methodology involves preprocessing raw data, feature extraction, and clustering similar data blocks together to reduce redundancy. We analyze key evaluation metrics: Get Storage Savings, Retrieval Time Improvement, and Scalability. K-means resulted in a 30<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${{\% }}$</tex> improvement in retrieval times while achieving the highest storage reduction of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%$25{{\% }}$</tex>. But DBSCAN handled noisy data well and hierarchical clustering provided granular relationships at the cost of high computational cost. We have demonstrated scalability and real-world practical application of this approach in such large-scale data environments with which we can perform sustainable data storage cost optimization as well as improved data retrieval performance in distributed systems. Future work will also examine dynamic clustering techniques as well as hybrid models that can improve system performance and adaptability even further.