Author name cluster

Jian Dong

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

AAAI Conference 2026 Conference Paper

Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory

Hongli Zhou
Hui Huang
Ziqing Zhao
Lvyuan Han
Huicheng Wang
Kehai Chen
Muyun Yang
Wei Bao

The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference.

PDF Details DOI

EAAI Journal 2024 Journal Article

A threat assessment method based on correlation-similarity information and three-way decisions in interval intuitionistic fuzzy environment

Jinhuan Zhang
Yujie Shan
Jian Dong

Threat assessment of targets is a critical component of combat decision-making. Existing threat assessment methods only consider specific dimensions of information. When information is missing or abnormal, they may fail to provide accurate assessment results. Moreover, most methods can only obtain the threat level of targets without providing target threat levels based on the battlefield environment. Therefore, in the context of interval intuitionistic fuzziness, this paper proposes a threat assessment method based on Grey Correlation Analysis and Technique for Order Preference by Similarity to an Ideal Solution (GCA-TOPSIS) and three-way decisions. This method utilizes interval intuitionistic fuzzy entropy to calculate attribute weights. It simultaneously considers information on both similarity and correlation between targets. By introducing grey correlation coefficients to balance the similarity and correlation between targets and the ideal solution, the method computes threat degrees based on the information from both dimensions. Through fuzzy evaluation information, it constructs loss functions, calculates decision thresholds, and thereby determines the threat levels of the targets. This approach is better suited for the complex and dynamic battlefield environment, providing decision-makers with target threat levels. Finally, we conducted simulation experiments under both complete data and data anomaly conditions. The results indicate that when the data noise reaches 0. 5 dB, the absolute error of GCA-TOPSIS is only 0. 08. When 66. 67% of the data is missing, the error is 0. 16. In both scenarios, the error is much smaller than that of other threat assessment methods. This demonstrates the method's strong robustness and its suitability for complex battlefields.

Details DOI

AAAI Conference 2023 Conference Paper

Decision-Making Context Interaction Network for Click-Through Rate Prediction

Xiang Li
Shuwei Chen
Jian Dong
Jin Zhang
Yongkang Wang
Xingxing Wang
Dong Wang

Click-through rate (CTR) prediction is crucial in recommendation and online advertising systems. Existing methods usually model user behaviors, while ignoring the informative context which influences the user to make a click decision, e.g., click pages and pre-ranking candidates that inform inferences about user interests, leading to suboptimal performance. In this paper, we propose a Decision-Making Context Interaction Network (DCIN), which deploys a carefully designed Context Interaction Unit (CIU) to learn decision-making contexts and thus benefits CTR prediction. In addition, the relationship between different decision-making context sources is explored by the proposed Adaptive Interest Aggregation Unit (AIAU) to improve CTR prediction further. In the experiments on public and industrial datasets, DCIN significantly outperforms the state-of-the-art methods. Notably, the model has obtained the improvement of CTR+2.9%/CPM+2.1%/GMV+1.5% for online A/B testing and served the main traffic of Meituan Waimai advertising system.

PDF Details DOI