Arrow Research search

Author name cluster

Pengfei Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

34 papers
2 author rows

Possible papers

34

AAAI Conference 2026 Short Paper

Efficient Preference Alignment via Pareto Exploration (Student Abstract)

  • Pengfei Liu
  • Rui Kong
  • Zongzhang Zhang

Hand-craft reward engineering requires domain knowledge with numerous trials and errors, while Preference-based Reinforcement Learning (PbRL) avoids manual reward design but often suffers from limited interpretability and unstable training. To address these issues, we propose a novel preference alignment framework. Our approach leverages large language models to generate sub-reward functions informed by prior knowledge and further align human preferences by optimizing the weights combining these sub-rewards. For policy learning, we introduce Policy Optimization via Pareto Regularization (POPR) which regularizes updates along Pareto-optimal directions. Experiments show that our framework improves reward quality and policy stability, achieving superior performance to expert-designed rewards across most tasks.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling

  • Yang Xiao
  • Chunpu Xu
  • Ruifeng Yuan
  • Jessie Wang
  • Wenjie Li
  • Pengfei Liu

Test-time compute scaling has emerged as a powerful paradigm for enhancing mathematical reasoning in large language models (LLMs) by allocating additional computational resources during inference. However, current methods employ uniform resource distribution across all reasoning sub-problems, creating fundamental bottlenecks where challenging sub-problems receive insufficient attention while routine operations consume disproportionate resources. This uniform allocation creates performance bottlenecks where additional computational resources yield diminishing returns. Inspired by dual-process theory, we propose SCALE (Selective Resource Allocation), a framework that selectively allocates computational resources based on sub-problem difficulty. SCALE operates through four stages: (1) problem decomposition into sequential reasoning sub-problems, (2) difficulty assessment of each sub-problem to distinguish between routine operations and computationally challenging sub-problems, (3) selective processing mode assignment between System 1 for simple sub-problems and System 2 for complex ones, and (4) sequential execution with context propagation. By concentrating resources on challenging sub-problems while processing routine operations efficiently, SCALE achieves substantial performance improvements with superior resource utilization. Extensive experiments demonstrate that SCALE significantly outperforms uniform scaling baselines, achieving accuracy improvements of up to 13.75 percentage points (57.50% to 71.25% on AIME25) while reducing computational costs by 33-53%, representing a major advance in test-time scaling that addresses fundamental limitations of current approaches.

PDF Details DOI

EAAI Journal 2025 Journal Article

A self-feedback knowledge elicitation approach for chemical reaction predictions

  • Pengfei Liu
  • Jun Tao
  • Zhixiang Ren

The task of chemical reaction predictions (CRPs) plays a pivotal role in advancing drug discovery and material science. However, while artificial intelligence (AI) has significantly advanced in capturing chemical reaction patterns, its effectiveness is still constrained by the vast and uncertain chemical reaction space and challenges in achieving high reaction selectivity, particularly due to existing methods’ limitations in fully leveraging the inherent knowledge within the data. To address these challenges, we introduce a data-curated self-feedback knowledge elicitation approach. This method starts from iterative optimization of molecular representations and facilitates the extraction of knowledge on chemical reaction types (RTs). Then, we employ adaptive prompt learning to infuse the prior knowledge into the large language model (LLM). As a result, we achieve significant enhancements: a 14. 2% increase in retrosynthesis prediction accuracy, a 74. 2% rise in reagent prediction accuracy, and an expansion in the model’s capability for handling multi-task chemical reactions. This research offers a novel paradigm for knowledge elicitation in scientific research and showcases the untapped potential of LLMs in CRPs.

Details DOI

EAAI Journal 2025 Journal Article

Artificial neural networks for finger vein recognition: A survey

  • Yimin Yin
  • Renye Zhang
  • Pengfei Liu
  • Wanxia Deng
  • Dayu Hu
  • Siliang He
  • Chen Li
  • Jinghua Zhang

Finger vein recognition is an emerging biometric recognition technology. Different from the other biometric features on the body surface, the venous vascular tissue of the fingers is buried deep inside the skin. Due to this advantage, finger vein recognition is highly stable and private. Finger veins are virtually impossible to steal and difficult to interfere with by external conditions. Unlike the finger vein recognition methods based on traditional machine learning, the artificial neural network technique, especially deep learning, does not rely on feature engineering and has superior performance. To summarize the development of finger vein recognition based on artificial neural networks, this paper collects 174 related papers. First, we introduce the background of finger vein recognition and the motivation for this survey. Then, the development history of artificial neural networks and the representative networks on finger vein recognition tasks are introduced. The public datasets widely used in finger vein recognition are then described. After that, we summarize the related finger vein recognition tasks based on classical neural networks and deep neural networks, respectively. Finally, the challenges and potential development directions in finger vein recognition are discussed. This paper provides a comprehensive and novel summary of the application of artificial neural networks in the finger vein recognition field.

Details DOI

AAAI Conference 2025 Conference Paper

Evaluating Mathematical Reasoning Beyond Accuracy

  • Shijie Xia
  • Xuefeng Li
  • Yixin Liu
  • Tongshuang Wu
  • Pengfei Liu

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs validity and redundancy to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. We explore different design options for the LLM-based evaluators and empirically demonstrate that ReasonEval, when instantiated with base models possessing strong mathematical knowledge and trained with high-quality labeled data, consistently outperforms baseline methods in the meta-evaluation datasets. We also highlight the strong generalization capabilities of ReasonEval. By utilizing ReasonEval to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We open-source the best-performing model, meta-evaluation script, and all evaluation results to facilitate future research.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scaling

  • Yang Xiao
  • Jiashuo WANG
  • Ruifeng Yuan
  • Chunpu Xu
  • Kaishuai Xu
  • Wenjie Li
  • Pengfei Liu

Large language models (LLMs) have demonstrated remarkable reasoning capabilities through test-time scaling approaches, particularly when fine-tuned with chain-of-thought (CoT) data distilled from more powerful large reasoning models (LRMs). However, these reasoning chains often contain verbose elements that mirror human problem-solving, categorized as progressive reasoning (the essential solution development path) and functional elements (verification processes, alternative solution approaches, and error corrections). While progressive reasoning is crucial, the functional elements significantly increase computational demands during test-time inference. We introduce PIR (Perplexity-based Importance Refinement), a principled framework that quantitatively evaluates the importance of each reasoning step based on its impact on answer prediction confidence. PIR systematically identifies and selectively prunes only low-importance functional steps while preserving all progressive reasoning components, creating optimized training data that maintains the integrity of the core solution path while reducing verbosity. Models fine-tuned on PIR-optimized data exhibit superior test-time scaling properties, generating more concise reasoning chains while achieving improved accuracy (+0. 9\% to +6. 6\%) with significantly reduced token usage (-3\% to -41\%) across challenging reasoning benchmarks (AIME, AMC, and GPQA Diamond). Our approach demonstrates strong generalizability across different model sizes, data sources, and token budgets, offering a practical solution for deploying reasoning-capable LLMs in scenarios where efficient test-time scaling, response time, and computational efficiency are valuable constraints. Code and dataset are available at the LIMOPro GitHub repository.

PDF Details

NeurIPS Conference 2025 Conference Paper

On Evaluating LLM Alignment by Evaluating LLMs as Judges

  • Yixin Liu
  • Pengfei Liu
  • Arman Cohan

Alignment with human preferences is an important evaluation aspect of LLMs, requiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong LLM judges. Conversely, LLMs themselves have also been extensively evaluated as judges for assessing alignment. In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evaluation consistency (GE-consistency) among various LLMs, revealing a strong correlation between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle (GPT-4o). Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evaluators. Our evaluation shows that our proposed benchmark, AlignEval, matches or surpasses widely used automatic LLM evaluation benchmarks, such as AlpacaEval and Arena-Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and evaluation capabilities, and introduces a benchmark that assesses alignment without directly evaluating model outputs.

PDF Details

AAAI Conference 2025 Conference Paper

RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance

  • Chengrui Wang
  • Pengfei Liu
  • Min Zhou
  • Ming Zeng
  • Xubin Li
  • Tiezheng Ge
  • Bo Zheng

Although diffusion models can generate high-quality human images, their applications are limited by the instability in generating hands with correct structures. In this paper, we introduce RHanDS, a conditional diffusion-based framework designed to refine malformed hands by utilizing decoupled structure and style guidance. The hand mesh reconstructed from the malformed hand offers structure guidance for correcting the structure of the hand, while the malformed hand itself provides style guidance for preserving the style of the hand. To alleviate the mutual interference between style and structure guidance, we introduce a two-stage training strategy and build a series of multi-style hand datasets. In the first stage, we use paired hand images for training to ensure stylistic consistency in hand refining. In the second stage, various hand images generated based on human meshes are used for training, enabling the model to gain control over the hand structure. Experimental results demonstrate that RHanDS can effectively refine hand structure while preserving consistency in hand style.

PDF Details DOI

ICLR Conference 2025 Conference Paper

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

  • Wenhong Zhu
  • Zhiwei He 0002
  • Xiaofeng Wang
  • Pengfei Liu
  • Rui Wang 0015

Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible. The code is available at https://github.com/zwhong714/weak-to-strong-preference-optimization.

Details

NeurIPS Conference 2024 Conference Paper

Alignment for Honesty

  • Yuqing Yang
  • Ethan Chern
  • Xipeng Qiu
  • Graham Neubig
  • Pengfei Liu

Recent research has made significant strides in aligning large language models (LLMs) with helpfulness and harmlessness. In this paper, we argue for the importance of alignment for \emph{honesty}, ensuring that LLMs proactively refuse to answer questions when they lack knowledge, while still not being overly conservative. However, a pivotal aspect of alignment for honesty involves discerning an LLM's knowledge boundaries, which demands comprehensive solutions in terms of metric development, benchmark creation, and training methodologies. We address these challenges by first establishing a precise problem definition and defining ``honesty'' inspired by the Analects of Confucius. This serves as a cornerstone for developing metrics that effectively measure an LLM's honesty by quantifying its progress post-alignment. Furthermore, we introduce a flexible training framework which is further instantiated by several efficient fine-tuning techniques that emphasize honesty without sacrificing performance on other tasks. Our extensive experiments reveal that these aligned models show a marked increase in honesty, as indicated by our proposed metrics. We open-source all relevant resources to facilitate future research at \url{https: //github. com/GAIR-NLP/alignment-for-honesty}.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

MathPile: A Billion-Token-Scale Pretraining Corpus for Math

  • Zengzhi Wang
  • Xuefeng Li
  • Rui Xia
  • Pengfei Liu

High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9. 5 billion tokens. Throughout its creation, we adhered to the principle of “less is more”, firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models’ mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

  • Zhen Huang
  • Zengzhi Wang
  • Shijie Xia
  • Xuefeng Li
  • Haoyang Zou
  • Ruijie Xu
  • Run-Ze Fan
  • Lyumanshan Ye

The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i. e. , AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11, 163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39. 97\% overall accuracy (28. 67\% for mathematics and 29. 71\% for physics), illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

  • Dongyu Ru
  • Lin Qiu
  • Xiangkun Hu
  • Tianhang Zhang
  • Peng Shi
  • Shuaichen Chang
  • Cheng Jiayang
  • Cunxiang Wang

Despite Retrieval-Augmented Generation (RAG) has shown promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems.

PDF Details DOI

YNICL Journal 2024 Journal Article

Unveiling MRI markers for Parkinson’s Disease: GABAergic dysfunction and cortical changes

  • Yuan Tian
  • Sijia Geng
  • Tianyi Liu
  • Qi Wang
  • Jianxiu Lian
  • Liangjie Lin
  • Jiayu Li
  • Tao Gong

OBJECTIVE: The study aimed to investigate changes in basal levels of the inhibitory γ-aminobutyric acid (GABA) neurotransmitter in the sensorimotor cortex (SMC) and cortical gyrification in patients with Parkinson's disease (PD), which could further identify potential imaging biomarkers for PD, particularly in patients with early-onset Parkinson's disease (EOPD). METHOD: Fifty patients with PD (EOPD: 10, late-onset Parkinson's disease [LOPD]: 40) and fifty-two age- and gender-matched healthy controls (HC) underwent GABA-edited 1H MRS of the SMC and high-resolution 3D T1-weighted brain imaging. GABA levels and local gyrification index (LGI) were calculated to assess GABAergic and cortical gyrification deficits in PD. RESULT: The Pearson correlation coefficients revealed significant negative associations between eight indicators, including GABA/Cr level and local gyrification index (LGI) of specific cortical regions (precentral, postcentral, entorhinal, superiortemporal, posteriorcingulate, cuneus, and transversetemporal cortex), and the likelihood of Parkinson's disease (r < -0.4, p < 0.001). Additionally, GABA levels were significantly lower in the SMC region of both EOPD and LOPD patients compared to healthy controls (mean ± SD [u.i.]: EOPD=0.081 ± 0.022 vs. Young-HC=0.112 ± 0.021, p = 0.003; LOPD=0.054 ± 0.024 vs. Old-HC=0.099 ± 0.021, p < 0.001). The logistic regression model was established by using multivariate analysis, identifying two statistically significant indicators: GABA/Cr and LGI of the transversetemporal. The combined model exhibited the highest AUC values in both younger and older populations. CONCLUSION: GABAergic dysfunction may play an important role in the pathogenesis of PD patients. Changes in neurotransmitter and morphological may serve as potential markers for the preclinical diagnosis and progression of PD, including EOPD.

Details DOI

NeurIPS Conference 2023 Conference Paper

FELM: Benchmarking Factuality Evaluation of Large Language Models

  • Shiqi Chen
  • Yiran Zhao
  • Jinghan Zhang
  • I-Chun Chern
  • Siyang Gao
  • Pengfei Liu
  • Junxian He

Assessing factuality of text generated by large language models (LLMs) is an emerging yet crucial research area, aimed at alerting users to potential errors and guiding the development of more reliable LLMs. Nonetheless, the evaluators assessing factuality necessitate suitable evaluation themselves to gauge progress and foster advancements. This direction remains under-explored, resulting in substantial impediments to the progress of factuality evaluators. To mitigate this issue, we introduce a benchmark for Factuality Evaluation of large Language Models, referred to as FELM. In this benchmark, we collect responses generated from LLMs and annotate factuality labels in a fine-grained manner. Contrary to previous studies that primarily concentrate on the factuality of world knowledge (e. g. information from Wikipedia), FELM focuses on factuality across diverse domains, spanning from world knowledge to math and reasoning. Our annotation is based on text segments, which can help pinpoint specific factual errors. The factuality annotations are further supplemented by predefined error types and reference links that either support or contradict the statement. In our experiments, we investigate the performance of several LLM-based factuality evaluators on FELM, including both vanilla LLMs and those augmented with retrieval mechanisms and chain-of-thought processes. Our findings reveal that while retrieval aids factuality evaluation, current LLMs are far from satisfactory to faithfully detect factual errors.

PDF Details

EAAI Journal 2023 Journal Article

LiDAR-camera fusion: Dual transformer enhancement for 3D object detection

  • Mu Chen
  • Pengfei Liu
  • Huaici Zhao

Recently, the progress in autonomous driving tries to leverage the strong complementarity of LiDAR point clouds and RGB images to realize a high-efficient 3D object detection task. However, some works just simply decorate the raw point clouds or point-cloud features with camera clues in a hard way, which cannot fully exploit the relevance between the two-modal data. In this paper, we propose a dual-feature interaction module that adopts a soft-fusion strategy to give guidance for the LiDAR-camera feature fusion by interacting the LiDAR and camera features with Transformer. Compared with the hard-fusion method, this soft-fusion method can decorate the LiDAR feature with a reliable image feature. Additionally, we design an uncertainty-based 3D Intersection over Union (IoU) metric in the training process. This strategy aims at modeling the unreliability of 3D IoU scores to alleviate the bad effects caused by the coupling problem of 3D properties. Experiments conducted on the KITTI dataset achieve significant improvements in the 3D object detection and bird’s eye view tasks when compared with the previous arts. Especially for the task of 3D object detection, our approach obtains 0. 68 and 0. 45 gains for the metric of A P 3 D on the moderate level and hard level, respectively.

Details DOI

NeurIPS Conference 2023 Conference Paper

LIMA: Less Is More for Alignment

  • Chunting Zhou
  • Pengfei Liu
  • Puxin Xu
  • Srinivasan Iyer
  • Jiao Sun
  • Yuning Mao
  • Xuezhe Ma
  • Avia Efrat

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1, 000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43\% of cases; this statistic is as high as 58\% when compared to Bard and 65\% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

PDF Details

JAIR Journal 2022 Journal Article

Can We Automate Scientific Reviewing?

  • Weizhe Yuan
  • Pengfei Liu
  • Graham Neubig

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing? ”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task. We then focus on the machine learning domain and collect a dataset of papers in the domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers as input and generate reviews as output. Comprehensive experimental results on the test set show that while system-generated reviews are comprehensive, touching upon more aspects of the paper than human-written reviews, the generated texts are less constructive and less factual than human-written reviews for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. Given these results, we pose eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research in this direction. We make relevant resource publicly available for use by future research: https://github. com/neulab/ReviewAdvisor. In addition, while our conclusion is that the technology is not yet ready for use in high-stakes review settings we provide a system demo, ReviewAdvisor (http://review.nlpedia.ai/), showing the current capabilities and failings of state-of-the-art NLP models at this task (see demo screenshot in A.2). A review of this paper written by the system proposed in this paper can be found in A.1.

PDF Details DOI

IROS Conference 2022 Conference Paper

Fidelity Evaluation of Virtual Traffic Based on Anomalous Trajectory Detection

  • Chaoneng Li
  • Qianwen Chao
  • Guanwen Feng
  • Qiongyan Wang
  • Pengfei Liu
  • Yunan Li 0001
  • Qiguang Miao

Measuring the fidelity of synthesized virtual traffic has become an important and fundamental concern for evaluating the performance of different traffic simulation techniques and applications of autonomous vehicle testing. In this work, we propose a novel method to evaluate the fidelity of any trajectory data from the perspective of anomalous trajectory detection. First, given the trajectory data to be evaluated as input, the method learns spatio-temporal traffic features and reconstructs the input trajectory through a Long Short-Term Memory (LSTM)-based autoencoder architecture. Then, the anomalous trajectories are detected by comparing the reconstructed trajectories and the input ones using the reconstruction error as the benchmark. Our method can detect eight different kinds of anomalous trajectory in terms of changes in velocity and moving direction. In order to evaluate the fidelity of the input trajectory, we design a perceptual evaluation on virtual traffic fidelity and derive a mapping from the reconstruction error to the evaluation score. We demonstrated the effectiveness and robustness of our metric through many experiments on real-world and synthetic trajectory data containing different types of motion anomalies.

Details

IJCAI Conference 2022 Conference Paper

I²R-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation

  • Yiwei Ding
  • Wenjin Deng
  • Yinglin Zheng
  • Pengfei Liu
  • Meihong Wang
  • Xuan Cheng
  • Jianmin Bao
  • Dong Chen

In this paper, we present the Intra- and Inter-Human Relation Networks I²R-Net for Multi-Person Pose Estimation. It involves two basic modules. First, the Intra-Human Relation Module operates on a single person and aims to capture Intra-Human dependencies. Second, the Inter-Human Relation Module considers the relation between multiple instances and focuses on capturing Inter-Human interactions. The Inter-Human Relation Module can be designed very lightweight by reducing the resolution of feature map, yet learn useful relation information to significantly boost the performance of the Intra-Human Relation Module. Even without bells and whistles, our method can compete or outperform current competition winners. We conduct extensive experiments on COCO, CrowdPose, and OCHuman datasets. The results demonstrate that the proposed model surpasses all the state-of-the-art methods. Concretely, the proposed method achieves 77. 4% AP on CrowPose dataset and 67. 8% AP on OCHuman dataset respectively, outperforming existing methods by a large margin. Additionally, the ablation study and visualization analysis also prove the effectiveness of our model.

PDF Details DOI

AAAI Conference 2022 Conference Paper

KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-training

  • Weizhe Yuan
  • Pengfei Liu

The surge in the number of scientific submissions has brought challenges to the work of peer review. In this paper, as a first step, we explore the possibility of designing an automated system, which is not meant to replace humans, but rather provide a first-pass draft for a machine-assisted human review process. Specifically, we present an end-to-end knowledge-guided review generation framework for scientific papers grounded in cognitive psychology research that a better understanding of text requires different types of knowledge. In practice, we found that this seemingly intuitive idea suffered from training difficulties. In order to solve this problem, we put forward an oracle pre-training strategy, which can not only make the KID-REVIEW better educated but also make the generated review cover more aspects. Experimentally, we perform a comprehensive evaluation (human and automatic) from different perspectives. Empirical results have shown the effectiveness of different types of knowledge as well as oracle pre-training. We make all code, relevant datasets available: https: //github. com/yyy-Apple/KIDReview as well as the KID- REVIEW system: http: //nlpeer. reviews.

PDF Details

NeurIPS Conference 2021 Conference Paper

BARTScore: Evaluating Generated Text as Text Generation

  • Weizhe Yuan
  • Graham Neubig
  • Pengfei Liu

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e. g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e. g. , machine translation, text summarization) and 7 different perspectives (e. g. , informativeness, factuality). Code to calculate BARTScore is available at https: //github. com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http: //explainaboard. nlpedia. ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.

PDF Details

AAAI Conference 2020 Conference Paper

Learning Sparse Sharing Architectures for Multiple Tasks

  • Tianxiang Sun
  • Yunfan Shao
  • Xiaonan Li
  • Pengfei Liu
  • Hang Yan
  • Xipeng Qiu
  • Xuanjing Huang

Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanism depends on the relations among the tasks, which is not easy since it is difficult to understand the underlying shared factors among these tasks. In this paper, we propose a novel parameter sharing mechanism, named Sparse Sharing. Given multiple tasks, our approach automatically finds a sparse sharing structure. We start with an over-parameterized base network, from which each task extracts a subnetwork. The subnetworks of multiple tasks are partially overlapped and trained in parallel. We show that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework. We conduct extensive experiments on three sequence labeling tasks. Compared with single-task models and three typical multi-task learning baselines, our proposed approach achieves consistent improvement while requiring fewer parameters.

PDF Details

AAAI Conference 2020 Conference Paper

Multi-Scale Self-Attention for Text Classification

  • Qipeng Guo
  • Xipeng Qiu
  • Pengfei Liu
  • Xiangyang Xue
  • Zheng Zhang

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi- Scale Transformer which uses multi-scale multi-head selfattention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

PDF Details

AAAI Conference 2020 Conference Paper

Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study

  • Jinlan Fu
  • Pengfei Liu
  • Qi Zhang

While neural network-based models have achieved impressive performance on a large body of NLP tasks, the generalization behavior of different models remains poorly understood: Does this excellent performance imply a perfect generalization model, or are there still some limitations? In this paper, we take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives and characterize the differences of their generalization abilities through the lens of our proposed measures, which guides us to better design models and training methods. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models in terms of breakdown performance analysis, annotation errors, dataset bias, and category relationships, which suggest directions for improvement. We have released the datasets: (ReCoNLL, PLONER) for the future research at our project page: http: //pfliu. com/InterpretNER/.

PDF Details

AAAI Conference 2020 Conference Paper

Zero-Shot Text-to-SQL Learning with Auxiliary Task

  • Shuaichen Chang
  • Pengfei Liu
  • Yun Tang
  • Jing Huang
  • Xiaodong He
  • Bowen Zhou

Recent years have seen great success in the use of neural seq2seq models on the text-to-SQL task. However, little work has paid attention to how these models generalize to realistic unseen data, which naturally raises a question: does this impressive performance signify a perfect generalization model, or are there still some limitations? In this paper, we first diagnose the bottleneck of the text-to- SQL task by providing a new testbed, in which we observe that existing models present poor generalization ability on rarely-seen data. The above analysis encourages us to design a simple but effective auxiliary task, which serves as a supportive model as well as a regularization term to the generation task to increase the models generalization. Experimentally, We evaluate our models on a large text-to-SQL dataset WikiSQL. Compared to a strong baseline coarse-to- fine model, our models improve over the baseline by more than 3% absolute in accuracy on the whole dataset. More interestingly, on a zero-shot subset test of WikiSQL, our models achieve 5% absolute accuracy gain over the baseline, clearly demonstrating its superior generalizability.

PDF Details

AAAI Conference 2019 Conference Paper

Contextualized Non-Local Neural Networks for Sequence Learning

  • Pengfei Liu
  • Shuaichen Chang
  • Xuanjing Huang
  • Jian Tang
  • Jackie Chi Kit Cheung

Recently, a large number of neural mechanisms and models have been proposed for sequence learning, of which selfattention, as exemplified by the Transformer model, and graph neural networks (GNNs) have attracted much attention. In this paper, we propose an approach that combines and draws on the complementary strengths of these two methods. Specifically, we propose contextualized non-local neural networks (CN3 ), which can both dynamically construct a task-specific structure of a sentence and leverage rich local dependencies within a particular neighbourhood. Experimental results on ten NLP tasks in text classification, semantic matching, and sequence labelling show that our proposed model outperforms competitive baselines and discovers task-specific dependency structures, thus providing better interpretability to users.

PDF Details

AAAI Conference 2019 Conference Paper

Learning Multi-Task Communication with Message Passing for Sequence Learning

  • Pengfei Liu
  • Jie Fu
  • Yue Dong
  • Xipeng Qiu
  • Jackie Chi Kit Cheung

We present two architectures for multi-task learning with neural sequence models. Our approach allows the relationships between different tasks to be learned dynamically, rather than using an ad-hoc pre-defined structure as in previous work. We adopt the idea from message-passing graph neural networks, and propose a general graph multi-task learning framework in which different tasks can communicate with each other in an effective and interpretable way. We conduct extensive experiments in text classification and sequence labelling to evaluate our approach on multi-task learning and transfer learning. The empirical results show that our models not only outperform competitive baselines, but also learn interpretable and transferable patterns across tasks.

PDF Details

AAAI Conference 2018 Conference Paper

Meta Multi-Task Learning for Sequence Modeling

  • Junkun Chen
  • Xipeng Qiu
  • Pengfei Liu
  • Xuanjing Huang

Semantic composition functions have been playing a pivotal role in neural representation learning of text sequences. In spite of their success, most existing models suffer from the underfitting problem: they use the same shared compositional function on all the positions in the sequence, thereby lacking expressive power due to incapacity to capture the richness of compositionality. Besides, the composition functions of different tasks are independent and learned from scratch. In this paper, we propose a new sharing scheme of composition function across multiple tasks. Specifically, we use a shared meta-network to capture the meta-knowledge of semantic composition and generate the parameters of the taskspecific semantic composition models. We conduct extensive experiments on two types of tasks, text classification and sequence tagging, which demonstrate the benefits of our approach. Besides, we show that the shared meta-knowledge learned by our proposed model can be regarded as off-theshelf knowledge and easily transferred to new tasks.

PDF Details

IJCAI Conference 2017 Conference Paper

Adaptive Semantic Compositionality for Sentence Modelling

  • Pengfei Liu
  • Xipeng Qiu
  • Xuanjing Huang

Representing a sentence with a fixed vector has shown its effectiveness in various NLP tasks. Most of the existing methods are based on neural network, which recursively apply different composition functions to a sequence of word vectors thereby obtaining a sentence vector. A hypothesis behind these approaches is that the meaning of any phrase can be composed of the meanings of its constituents. However, many phrases, such as idioms, are apparently non-compositional. To address this problem, we introduce a parameterized compositional switch, which outputs a scalar to adaptively determine whether the meaning of a phrase should be composed of its two constituents. We evaluate our model on five datasets of sentiment classification and demonstrate its efficacy with qualitative and quantitative experimental analysis.

PDF Details

IJCAI Conference 2017 Conference Paper

Dynamic Compositional Neural Networks over Tree Structure

  • Pengfei Liu
  • Xipeng Qiu
  • Xuanjing Huang

Tree-structured neural networks have proven to be effective in learning semantic representations by exploitingsyntactic information. In spite of their success, most existing models suffer from the underfitting problem: they recursively use the same shared compositional function throughout the whole compositional process and lack expressive power due to inability to capture the richness of compositionality. In this paper, we address this issue by introducing the dynamic compositional neural networks over tree structure (DC-TreeNN), in which the compositional function is dynamically generated by a meta network. The role of meta-network is to capture the metaknowledge across the different compositional rules and formulate them. Experimental results on two typical tasks show the effectiveness of the proposed models.

PDF Details

AAAI Conference 2016 Conference Paper

Discourse Relations Detection via a Mixed Generative-Discriminative Framework

  • Jifan Chen
  • Qi Zhang
  • Pengfei Liu
  • Xuanjing Huang

Word embeddings, which can better capture the fine-grained semantics of words, have proven to be useful for a variety of natural language processing tasks. However, because discourse structures describe the relationships between segments of discourse, word embeddings cannot be directly integrated to perform the task. In this paper, we introduce a mixed generative-discriminative framework, in which we use vector offsets between embeddings of words to represent the semantic relations between text segments and Fisher kernel framework to convert a variable number of vector offsets into a fixed length vector. In order to incorporate the weights of these offsets into the vector, we also propose the Weighted Fisher Vector. Experimental results on two different datasets show that the proposed method without using manually designed features can achieve better performance on recognizing the discourse level relations in most cases.

PDF Details

IJCAI Conference 2016 Conference Paper

Recurrent Neural Network for Text Classification with Multi-Task Learning

  • Pengfei Liu
  • Xipeng Qiu
  • Xuanjing Huang

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.

PDF Details

IJCAI Conference 2015 Conference Paper

Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model

  • Pengfei Liu
  • Xipeng Qiu
  • Xuanjing Huang

Distributed word representations have a rising interest in NLP community. Most of existing models assume only one vector for each individual word, which ignores polysemy and thus degrades their effectiveness for downstream tasks. To address this problem, some recent work adopts multiprototype models to learn multiple embeddings per word type. In this paper, we distinguish the different senses of each word by their latent topics. We present a general architecture to learn the word and topic embeddings efficiently, which is an extension to the Skip-Gram model and can model the interaction between words and topics simultaneously. The experiments on the word similarity and text classification tasks show our model outperforms state-of-the-art methods.

PDF Details
v2026.09.13