Automatically Benchmarking LLM Code Agents through Agent-driven Annotation and Evaluation

Recent advances in code agents have enabled automated software development at the project level, supported by large language models(LLMs)andwidelyadoptedtools. However, existingbenchmarks for code agent evaluation face two major limitations: high annotation cost and expertise requirements, and rigid evaluation metrics thatrelyprimarilyonunittests. Toaddressthesechallenges, wepropose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse and challenging project-level tasks. Based on this approach, we introduce PRDBench, a novel benchmark comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Document (PRD) requirements, comprehensive evaluation criteria, and reference implementations. PRDBench features rich data sources, high task complexity, and flexible metrics. We further employ an Agent-as-a-Judge paradigm to score agent outputs, enabling the evaluation of various test types beyond unit tests. Extensive experiments on PRDBench demonstrate its effectiveness in assessing the capabilities of both code agents and evaluation agents, providing a scalable and robust framework for annotation and evaluation.

Authors

• Lingyue Fu Shanghai Jiao Tong University
• Bolun Zhang Shanghai Jiao Tong University
• Hao Guan Shanghai Jiao Tong University
• Yaoming Zhu Meituan
• Lin Qiu Meituan
• Weiwen Liu Shanghai Jiao Tong University
• Xuezhi Cao Meituan
• Xunliang Cai Meituan
• Weinan Zhang Shanghai Jiao Tong University
• Yong Yu Shanghai Jiao Tong University

Keywords

• Code Agent
• Agent Evaluation
• Large Language Models