Optimisation of distributed storage technology for large-scale data based on Hadoop technology

The fast growth of big data technology makes effective storage and processing of vast amounts of data a major concern in contemporary computing systems. The growing data scale results in specific bottlenecks in data storage, task scheduling, and resource allocation even in the conventional distributed systems. Thus, this work presents a large-scale data distributed storage optimisation model based on Hadoop, i.e., Hadoop-OptiStor, which intends to improve the efficiency of Hadoop in big data processing by optimising important technologies such data distribution and store copy management. By means of experimental verification, the Hadoop-OptiStor model presented in this work exhibits notable improvement in numerous important criteria and performs better than the conventional distributed system. The optimisation model has higher application possibilities and practical value, according to the testing results; it can also efficiently increase resource utilisation, lower compute and storage bottlenecks.