Adaptive Benchmarking for Data System using LLMs

Benchmarking is crucial for understanding the complexity of big data systems. However, designing and running benchmarks is often tedious, time-intensive, and can be costly where the large data volumes required for benchmarks drive up storage needs and costs.Moreover, standard benchmarks may not always reflect user requirements, as users might want to adjust data generation methods or introduce new query types. In this work, we leverage Large Language Models (LLMs) to transform user prompts into detailed benchmarking plans. These plans can: (1) load and configure data systems, (2) generate benchmark data, (3) execute benchmarks while monitoring runtime, and (4) adapt the next steps based on the gathered metrics and user prompt. We use DSPy framework to guarantee the generated plan adheres to the user and system requirements and optimize the LLM-based workflow.