Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study

Large Language Models (LLMs) hold promise in automating data analysis tasks, yet open-source models face significant limitations in these kinds of reasoning-intensive scenarios. In this work, we investigate strategies to enhance the data analysis capabilities of open-source LLMs. By curating a seed dataset of diverse, realistic scenarios, we evaluate models across three dimensions: data understanding, code generation, and strategic planning. Our analysis reveals three key findings: (1) Strategic planning quality serves as the primary determinant of model performance; (2) Interaction design and task complexity significantly influence reasoning capabilities; (3) Data quality demonstrates a greater impact than diversity in achieving optimal performance. We leverage these insights to develop a data synthesis methodology, demonstrating significant improvements in open-source LLMs' analytical reasoning capabilities.

Authors

• Yuqi Zhu Academy of Military Sciences, China
• Yi Zhong National University of Defense Technology
• Jintian Zhang Zhejiang University Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph
• Ziheng Zhang Jarvis Research Center, Tencent YouTu Lab
• Shuofei Qiao Zhejiang University
• Yujie Luo Zhejiang University Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph
• Lun Du Microsoft
• Da Zheng Ant Group Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph
• Ningyu Zhang Zhejiang University Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph
• Huajun Chen College of Computer Science and Technology, Zhejiang University ZJU-Ant Group Joint Lab of Knowledge Graph Zhejiang Key Laboratory of Big Data Intelligent Computing

Keywords

No keywords are indexed for this paper.