Data-driven Database Engineering at Snowflake

Snowflake’s cloud-native data platform processes billions of queries daily and scans petabytes of data, creating unique database engineering challenges. As a service-oriented platform, we rely on detailed query telemetry, workload replay, and staged rollouts to drive continuous engine innovation. This talk highlights two projects from our Berlin engineering team that exemplify this data-driven approach to database engineering. First, we examine how query telemetry and workload analysis informed the transformation of Snowflake’s analytical execution engine to handle transactional-scale throughput. Second, we explore the redesign of Snowflake’s dynamic join strategy, replacing a static method with a holistic, adaptive algorithm. Through rigorous testing, workload replay, and incremental rollouts, we ensured a smooth production transition for customers. By examining these case studies, we provide practical insights into the intersection of database research and large-scale system engineering. The lessons learned highlight how data-driven methods can effectively address complex engineering challenges in modern database systems, offering perspectives relevant to both academic researchers and industry practitioners.