Self-Correction Distillation for Structured Data Question Answering

Structured data question answering (QA), including table QA, Knowledge Graph (KG) QA, and temporal KG QA, is a pivotal research area. Advances in large language models (LLMs) have driven significant progress in unified structural QA frameworks like TrustUQA. However, these frameworks face challenges when applied to small-scale LLMs since small-scale LLMs are prone to errors in generating structured queries. To improve the structured data QA ability of small-scale LLMs, we propose a self-correction distillation (SCD) method. In SCD, an error prompt mechanism (EPM) is designed to detect errors and provide customized error messages during inference, and a two-stage distillation strategy is designed to transfer large-scale LLMs' query-generation and error-correction capabilities to small-scale LLM. Experiments across 5 benchmarks with 3 structured data types demonstrate that our SCD achieves the best performance and superior generalization on small-scale LLM (8B) compared to other distillation methods, and closely approaches the performance of GPT4 on some datasets. Furthermore, large-scale LLMs equipped with EPM surpass the state-of-the-art results on most datasets.

Authors

• Yushan Zhu Zhejiang University
• Wen Zhang
• Long Jin Lanzhou University
• Mengshu Sun Ant Group
• Ling Zhong Zhejiang Lab
• Zhiqiang Liu Zhejiang University
• Juan Li
• Lei Liang Ant Group
• Chong Long JIUTIAN Research
• Chao Deng Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, China School of Software, Tsinghua University, Beijing, China
• Junlan Feng China Mobile Research Beijing

Keywords

No keywords are indexed for this paper.