FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL
Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.
Authors
- Erick Lee
- Gaetano Rossiello (ORCID: https://orcid.org/0000-0003-1042-4782)
- Nandana Mihindukulasooriya (ORCID: https://orcid.org/0000-0003-1707-4842)
- Lisa D. Amini (ORCID: https://orcid.org/0000-0001-9097-0497)
- Alfio Gliozzo (ORCID: https://orcid.org/0000-0002-8044-2911)
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.48550/arxiv.2611.17093
- Primary Topic
- Advanced Database Systems and Queries
- Type
- preprint