A hierarchical workload-aware optimization framework for Cloud-Based Big Data Analytics Using Parquet and Delta Lake

Large enterprises have seen their data volumes grow quickly, making it hard to gain scalable, cost-effective and efficient analytic solutions with traditional data management infrastructures. In this paper, a hierarchical workload-aware optimization framework is proposed and empirically introduce test on cloud-based big data analytics system. To develop and experimentally validate proposed hierarchical workload-aware optimization framework, dynamically selects and configures storage format, compression, physical data organization, clustering strategy, and computational resources according to workload characteristics and system constraints.