WritingBench: A Comprehensive Benchmark for Generative Writing

Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.

Authors

• Yuning Wu Remin University of China
• Jiahao Mei
• Ming Yan Centre for Frontier AI Research (CFAR), A*STAR, Singapore
• Chenliang Li Wuhan University
• Shaopeng Lai
• Yuran Ren
• Zijia Wang School of Informatics, Xiamen University
• Ji Zhang Alibaba Group
• Mengyue Wu Shanghai Jiao Tong University
• Qin Jin Renmin University of China
• Fei Huang Tongyi Lab

Keywords

No keywords are indexed for this paper.