ScaleLLM: A Technique for Scalable LLM-augmented Data Systems

Large language models (LLMs) offer powerful semantic insights for data analytics, but row-by-row LLM calls quickly become prohibitively expensive in large datasets. We introduce ScaleLLM, a novel system that substantially reduces both latency and cost on text classification tasks. ScaleLLM couples LLM-generated labels on a small subset of data with a lightweight machine learning model for large-scale inference. This approach provides significant speed-ups-up to 37×-while maintaining accuracy close to that of a full LLM baseline, converging within 1% of its accuracy in several tasks. ScaleLLM also provides cost-accuracy trade-off projections, giving users fine-grained control over performance trade-offs. Our demonstration illustrates ScaleLLM's reusable embedding views, efficient inference architecture, and potential for integration with query optimization frameworks in LLM-augmented database systems.