JSONBench: A High-Quality Benchmark for Diverse JSON-Oriented Tasks

Large Language Models (LLMs) have shown remarkable capabilities in various data-oriented tasks, including handling structured (e.g., table) and unstructured data (e.g., text). However, their performance in handling semi-structured JSON data remains largely unexplored. To bridge this gap, we introduce JSONBench, a large high-quality benchmark for diverse JSON-oriented tasks consisting of 74,229 JSON documents from 5 sources (e.g., wargame, new_game) and 206,243 task instances across 6 task types (e.g., Information Search, Attribute Finding). To effectively construct such a benchmark, we propose a twostage framework, comprising JSON_Synthesis and JSON_Task_Synthesis. We design a unified quality evaluation criterion with two dimensions (accuracy and diversity) to score different data. Our experiments demonstrate that JSONBench provides more diverse JSON-oriented task instances compared to task datasets of single types. We also provide practical insights into how data mixing can enhance data quality. To our knowledge, JSONBench is the first benchmark for diverse JSONoriented tasks, facilitating future research in this area.