FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification

We introduce FIXME, the first end-to-end and large-scale benchmark for evaluating Large Language Models (LLMs) in hardware design functional verification (FV). Comprising 747 tasks derived from real-world hardware designs, FIXME spans five core FV sub-sets: specification comprehension, reference model generation, testbench generation, assertion design, and RTL debugging. To ensure high data quality, we developed an AI-human collaborative framework for agile data curation and annotation. This process resulted in 25,000 lines of verified RTL, 35,000 lines of enhanced testbenches, and over 1,200 SystemVerilog Assertions. Furthermore, through expert-guided optimization within the multi-agent aided flow, we achieved a remarkable 45.57% improvement in average functional coverage, underscoring the benchmark's robustness. Through evaluation of state-of-the-art LLMs like GPT-4.1, FIXME identifies key limitations and provides actionable insights, advancing the potential of LLM-driven automation in hardware design functional verification.

Authors

• Gwok-Waa Wan Southeast University, Nanjing, Jiangsu, China National Center of Technology Innovation for EDA, Nanjing, Jiangsu, China
• SamZaak Wong School of Integrated Circuits, Southeast University, Nanjing, China National Center of Technology Innovation for EDA, Nanjing, China
• Shengchu Su Southeast University, Nanjing, Jiangsu, China
• Chenxu Niu
• Ning Wang Huawei Inc.
• Xinlai Wan National Center of Technology Innovation for EDA, Nanjing, China
• Qixiang Chen National Center of Technology Innovation for EDA, Nanjing, Jiangsu, China
• Mengnv Xing National Center of Technology Innovation for EDA, Nanjing, Jiangsu, China
• Jingyi Zhang Fujian Key Laboratory of Sensing and Computing for Smart Cities, Xiamen University, China Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, China
• Jianmin Ye Southeast University, Nanjing, Jiangsu, China
• Yubo Wang
• Rongchang Song National Center of Technology Innovation for EDA, Nanjing, Jiangsu, China
• Tao Ni
• Qiang Xu
• Nan Guan Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China
• Zhe Jiang University of Florida
• Xi Wang
• Yong Chen Geely Automobile Research Institute
• Jun Yang Department of Automation, Tsinghua University

Keywords

No keywords are indexed for this paper.