EFFICIENT BIG DATA STORAGE SOLUTIONS FOR DISTRIBUTED CLOUD COMPUTING SYSTEMS

Storing large volumes of data in distributed cloud computing systems can cause a problem, as managing both kinds of data, structured or unstructured, is difficult with existing tools like the Hadoop Distributed File System (HDFS) and object storage systems. Existing solutions are not effective at managing both throughput, growth capacity, and affordability simultaneously, particularly for data workloads with a wide range of use patterns. Although HDFS is designed for ordered data and Amazon S3 handles a large amount of unstructured data, there are no solutions in these approaches to quickly access or allocate resources to different kinds of data. To address this gap, a new hybrid architecture is proposed that combines HDFS with Amazon S3 and allocates data according to its type and how often it is used. The aim of this research is to explore how it integrates different systems to improve data retrieval time, ensure fault tolerance, and make everything cost-efficient for both old and new data in the cloud. By measuring the systems using the KDD Cup 1999 and UNSW-NB15 datasets, it is found that the hybrid model retrieves data faster, more reliably, more scalable, and more economical than HDFS and object storage used alone. The results of this study describe a stable and practical solution for data storage in the cloud and set a path for new studies and practical cloud data storage improvements.