EAAI Journal 2026 Journal Article
ScholarLens: Tracking the growing penetration of Large Language Models in scholarly writing and peer review
- Li Zhou
- Ruijie Zhang
- Xunlian Dai
- Daniel Hershcovich
- Haizhou Li
Although the widespread use of Large Language Models (LLMs) brings convenience, it also raises concerns about the credibility of academic research and scholarly processes. To better understand the extent and characteristics of LLM use in scholarly writing and peer review, the penetration of LLMs across academic workflows is evaluated from multiple perspectives and dimensions, providing compelling evidence of their growing influence. A framework consisting of two components is proposed: ScholarLens, a curated dataset of human-written and LLM-generated content across scholarly writing and peer review for multi-perspective evaluation, and LLMetrica, a tool for assessing LLM penetration using rule-based metrics and model-based detectors for multi-dimensional evaluation. The effectiveness of LLMetrica is demonstrated through experiments, revealing the increasing role of LLMs in scholarly processes. These findings emphasize the need for transparency, accountability, and ethical practices in the use of LLMs to maintain academic credibility.