StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce $\textbf{StructEval}$, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: $\textbf{1)}$ generation tasks, producing structured output from natural language prompts, and $\textbf{2)}$ conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps—even state-of-the-art models like o1-mini achieve only $75.58$ average score, with open-source alternatives lagging approximately $10$ points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.

Authors

• Jialin Yang
• Dongfu Jiang
• Tony He
• Sherman Siu
• Yuxuan Zhang University of Science and Technology of China
• Disen Liao
• Zhuofeng Li
• Huaye Zeng
• Yiming Jia
• Haozhe Wang
• Benjamin Schneider
• Chi Ruan
• Wentao Ma Anhui Agricultural University
• Zhiheng Lyu
• Yifei Wang South China University of Technology
• Yi Lu
• Quy Duc Do
• Ziyan Jiang Zhejiang University Dalian University of Technology
• Ping Nie
• Wenhu Chen Google AI

Keywords

No keywords are indexed for this paper.