Exploring Zero-Shot Prompting for Generating Data Format Descriptions

Parsers validate and process untrusted user input and transform it into data structures that provide easier access. Software engineers either build these parsers for data formats from scratch or leverage libraries targeting specific formats. The recent surge in data description languages (DDLs) and parser combinator libraries for parsing data formats has aided developers in producing parsers using standardized tools. However, producing a parser for an unfamiliar data format in an unfamiliar DDL can be daunting, given the learning curve of understanding two specifications or manuals well. As more researchers adopt tools such as GitHub Copilot [39] to simplify their programming tasks, we ask whether LLMs already hold sufficient knowledge to produce valid DDL specifications for popular data formats. To explore this, we systematically prompt LLMs to provide specifications in valid DDL syntax and evaluate whether these specifications are syntactically valid and correct. We found that while some LLMs, such as GPT 4 Turbo, Claude 3.5 Sonnet, and Deepseek V3, can produce valid Kaitai Struct YAML files, Hammer C files, and Rust Nom files, they struggle to produce valid specifications in complex DDLs, such as DaeDaLus, Spicy, and DFDL. In general, all LLMs fare much better at producing syntactically valid C code using the Hammer library and Rust code using the Nom library, given the large corpora of valid C and Rust code available. None of the LLMs in our test were able to produce a valid DFDL file or DaeDaLus file. We also found that while providing the specification manuals for the DDLs did not help in producing more syntactically valid specifications, providing sample specification files modestly increased the number of successful compilations.