DuoSQL: towards elastic data warehousing via separated data management and processing

Abstract Moving data warehouses (DWs) to the cloud is what today’s companies consider a trend towards cost-effective data management. To fully achieve the goal, the cloud DW system is supposed to adjust its resource provisioning to adapt to changing workload requirements. However, traditional data warehousing architecture lacks the flexibility for on-demand resource control, which severely restricts cost optimization and quality of service for both cloud providers and users. To build cloud DWs, new architectures are needed. This paper explores an architecture that decouples data management and processing to enable on-demand resource control. This optimized design enhances system elasticity and adaptability. However, this separation design is not without cost, as cooperation overhead can be high if not well optimized. For proof of concept, we build a prototype system, DuoSQL, using PostgreSQL for data management and Spark for data processing. To optimize cooperation, we conduct joint parameter tuning to improve overall system performance. We validate the system with the TPC-H benchmark. Results show the decoupling approach is flexible and offers significant performance potential.