Xiaofeng He

AAAI Conference 2025 Conference Paper

Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit

• Qizhou Chen
• Taolin Zhang
• Chengyu Wang
• Xiaofeng He
• Dakan Wang
• Tingting Liu

Model editing aims to correct outdated or erroneous knowledge in large models without costly retraining. Recent research discovered that the mid-layer representation of the subject's final token in a prompt has a strong influence on factual predictions, and developed Large Language Model (LLM) editing techniques based on this observation. However, for Vision-LLMs (VLLMs), how visual representations impact the predictions from a decoder-only language model remains largely unexplored. To the best of our knowledge, model editing for VLLMs has not been extensively studied in the literature. In this work, we employ the contribution allocation and noise perturbation methods to measure the contributions of visual representations for token predictions. Our attribution analysis shows that visual representations in mid-to-later layers that are highly relevant to the prompt contribute significantly to predictions. Based on these insights, we propose *VisEdit*, a novel model editor for VLLMs that effectively corrects knowledge by editing intermediate visual representations in regions important to the edit prompt. We evaluated *VisEdit* using multiple VLLM backbones and public VLLM editing benchmark datasets. The results show the superiority of *VisEdit* over the strong baselines adapted from existing state-of-the-art editors for LLMs.

NeurIPS Conference 2025 Conference Paper

UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models

• Qizhou Chen
• Dakan Wang
• Taolin Zhang
• Zaoming Yan
• Chengsong You
• Chengyu Wang
• Xiaofeng He

Model editing aims to efficiently revise incorrect or outdated knowledge within LLMs without incurring the high cost of full retraining and risking catastrophic forgetting. Currently, most LLM editing datasets are confined to narrow knowledge domains and cover a limited range of editing evaluation. They often overlook the broad scope of editing demands and the diversity of ripple effects resulting from edits. In this context, we introduce \uniedit, a unified benchmark for LLM editing grounded in open-domain knowledge. First, we construct editing samples by selecting entities from 25 common domains across five major categories, utilizing the extensive triple knowledge available in open-domain knowledge graphs to ensure comprehensive coverage of the knowledge domains. To address the issues of generality and locality in editing, we design an Neighborhood Multi-hop Chain Sampling (NMCS) algorithm to sample subgraphs based on a given knowledge piece to entail comprehensive ripple effects to evaluate. Finally, we employ proprietary LLMs to convert the sampled knowledge subgraphs into natural language text, guaranteeing grammatical accuracy and syntactical diversity. Extensive statistical analysis confirms the scale, comprehensiveness, and diversity of our \uniedit benchmark. We conduct comprehensive experiments across multiple LLMs and editors, analyzing their performance to highlight strengths and weaknesses in editing across open knowledge domains and various evaluation criteria, thereby offering valuable insights for future research endeavors.

AAAI Conference 2024 Conference Paper

CIDR: A Cooperative Integrated Dynamic Refining Method for Minimal Feature Removal Problem

• Qian Chen
• Taolin Zhang
• Dongyang Li
• Xiaofeng He

The minimal feature removal problem in the post-hoc explanation area aims to identify the minimal feature set (MFS). Prior studies using the greedy algorithm to calculate the minimal feature set lack the exploration of feature interactions under a monotonic assumption which cannot be satisfied in general scenarios. In order to address the above limitations, we propose a Cooperative Integrated Dynamic Refining method (CIDR) to efficiently discover minimal feature sets. Specifically, we design Cooperative Integrated Gradients (CIG) to detect interactions between features. By incorporating CIG and characteristics of the minimal feature set, we transform the minimal feature removal problem into a knapsack problem. Additionally, we devise an auxiliary Minimal Feature Refinement algorithm to determine the minimal feature set from numerous candidate sets. To the best of our knowledge, our work is the first to address the minimal feature removal problem in the field of natural language processing. Extensive experiments demonstrate that CIDR is capable of tracing representative minimal feature sets with improved interpretability across various models and datasets.

ECAI Conference 2024 Conference Paper

R 4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Models

• Taolin Zhang 0001
• Dongyang Li
• Qizhou Chen
• Chengyu Wang 0001
• Longtao Huang
• Hui Xue 0001
• Xiaofeng He
• Jun Huang 0007

Retrieval-augmented large language models (LLMs) leverage relevant content retrieved by information retrieval systems to generate correct responses, aiming to alleviate the hallucination problem. However, existing retriever-responder methods typically append relevant documents to the prompt of LLMs to perform text generation tasks without considering the interaction of fine-grained structural semantics between the retrieved documents and the LLMs. This issue is particularly important for accurate response generation as LLMs tend to “lose in the middle” when dealing with input prompts augmented with lengthy documents. In this work, we propose a new pipeline named “Reinforced Retriever-Reorder-Responder” (R4) to learn document orderings for retrieval-augmented LLMs, thereby further enhancing their generation abilities while the large numbers of parameters of LLMs remain frozen. The reordering learning process is divided into two steps according to the quality of the generated responses: document order adjustment and document representation enhancement. Specifically, document order adjustment aims to organize retrieved document orderings into beginning, middle, and end positions based on graph attention learning, which maximizes the reinforced reward of response quality. Document representation enhancement further refines the representations of retrieved documents for responses of poor quality via document-level gradient adversarial learning. Extensive experiments demonstrate that our proposed pipeline achieves better factual question-answering performance on knowledge-intensive tasks compared to strong baselines across various public datasets. The source codes and trained models will be released upon paper acceptance.

IROS Conference 2022 Conference Paper

A 2D Georeferenced Map Aided Visual-Inertial System for Precise UAV Localization

• Jun Mao
• Lilian Zhang
• Xiaofeng He
• Hao Qu
• Xiaoping Hu

Precise geolocalization is crucial for unmanned aerial vehicles (UAVs). However, most current deployed UAVs rely on the global navigation satellite systems (GNSS) for geolocalization. In this paper, we propose to use a lightweight visual-inertial system with a 2D georeferenced map to obtain accurate geodetic positions for UAVs. The proposed system firstly integrates a micro inertial measurement unit (MIMU) and a monocular camera to build a visual-inertial odometry (VIO) to consecutively estimate the UAV's motion states and reconstruct the 3D position of the observed visual features in the local world frame. To obtain the geolocation, the visual features tracked by the odometry are further registered to the 2D georeferenced map. While most conventional methods perform image-level aerial image registration, we propose to align the reconstructed 3D points with the map, and then use the registered 3D points to relocalize the vehicle in the geodetic frame, which helps to improve the geolocalization accuracy. Finally, a pose graph is deployed to fuse the geolocation from the point registration and the local navigation result from the visual-inertial odometry (VIO) to obtain smooth and drift-free geolocalization results. We have validated the proposed method by installing the sensors to a UAV body rigidly and have conducted two real-world flights in different environments with unknown initials. The results show that the proposed method can achieve less than 4m position error in flight at about 100m high and less than 9m position error in flight at about 300m high.

AAAI Conference 2022 Conference Paper

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

• Taolin Zhang
• Chengyu Wang
• Nan Hu
• Minghui Qiu
• Chengguang Tang
• Xiaofeng He
• Jun Huang

Knowledge-Enhanced Pre-trained Language Models (KE- PLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities. Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. To guarantee effective knowledge injection, previous studies integrate models with knowledge encoders for representing knowledge retrieved from knowledge graphs. The operations for knowledge retrieval and encoding bring significant computational burdens, restricting the usage of such models in real-world applications that require high inference speed. In this paper, we propose a novel KEPLM named DKPLM that decomposes knowledge injection process of the pre-trained language models in pre-training, fine-tuning and inference stages, which facilitates the applications of KEPLMs in realworld scenarios. Specifically, we first detect knowledge-aware long-tail entities as the target for knowledge injection, enhancing the KEPLMs’ semantic understanding abilities and avoiding injecting redundant information. The embeddings of long-tail entities are replaced by “pseudo token representations” formed by relevant knowledge triples. We further design the relational knowledge decoding task for pre-training to force the models to truly understand the injected knowledge by relation triple reconstruction. Experiments show that our model outperforms other KEPLMs significantly over zeroshot knowledge probing tasks and multiple knowledge-aware language understanding tasks. We further show that DKPLM has a higher inference speed than other competing models due to the decomposing mechanism.

IROS Conference 2021 Conference Paper

A Bio-Inspired Multi-Sensor System for Robust Orientation and Position Estimation

• Jia Xie
• Xiaofeng He
• Jun Mao
• Lilian Zhang
• Guoliang Han
• Wenzhou Zhou
• Xiaoping Hu

The nature animals have evolved highly efficient and robust organs that support their complex daily navigation tasks. To mimic animal’s navigation capability, we present a novel bio-inspired navigation system that draws inspirations from nature animals in this paper. The system consists of a three-axis magnetometer, a monocular camera, a micro inertial measurement unit (MIMU) and a polarization camera. While dead reckoning, orientation, and landmark recognition are considered as three most important capability for various species, we also designed corresponding algorithms based on the bio-inspired sensing system to perform autonomous navigation. In detail, the dead reckoning component is accomplished by integrating the monocular camera and the MIMU into a visual inertial odometry (VIO) and the orientation capability is achieved by combining the absolute orientation from the magnetometer with the relative orientation from the VIO. A loop closure detection is then used as the landmark recognition component to reduce the navigation drifts. All the three components are fused with a graph optimization method to generate the robust navigation result. To valid the proposed navigation sensing system and the algorithms, we have conducted series of experiments on ground and aerial unmanned vehicles, and have added orientation noise to testify the accuracy and robustness of the system.

AAAI Conference 2021 Conference Paper

KEML: A Knowledge-Enriched Meta-Learning Framework for Lexical Relation Classification

• Chengyu Wang
• Minghui Qiu
• Jun Huang
• Xiaofeng He

Lexical relations describe how concepts are semantically related, in the form of relation triples. The accurate prediction of lexical relations between concepts is challenging, due to the sparsity of patterns indicating the existence of such relations. We propose the Knowledge-Enriched Meta-Learning (KEML) framework to address lexical relation classification. In KEML, the LKB-BERT (Lexical Knowledge Base-BERT) model is first presented to learn concept representations from text corpora, with rich lexical knowledge injected by distant supervision. A probabilistic distribution of auxiliary tasks is defined to increase the model’s ability to recognize different types of lexical relations. We further propose a neural classifier integrated with special relation recognition cells, in order to combine meta-learning over the auxiliary task distribution and supervised learning for LRC. Experiments over multiple datasets show KEML outperforms state-of-the-art methods.

AAAI Conference 2019 Conference Paper

Improving Hypernymy Prediction via Taxonomy Enhanced Adversarial Learning

• Chengyu Wang
• Xiaofeng He
• Aoying Zhou

Hypernymy is a basic semantic relation in computational linguistics that expresses the “is-a” relation between a generic concept and its specific instances, serving as the backbone in taxonomies and ontologies. Although several NLP tasks related to hypernymy prediction have been extensively addressed, few methods have fully exploited the large number of hypernymy relations in Web-scale taxonomies. In this paper, we introduce the Taxonomy Enhanced Adversarial Learning (TEAL) for hypernymy prediction. We first propose an unsupervised measure U-TEAL to distinguish hypernymy with other semantic relations. It is implemented based on a word embedding projection network distantly trained over a taxonomy. To address supervised hypernymy detection tasks, the supervised model S-TEAL and its improved version, the adversarial supervised model AS-TEAL, are further presented. Specifically, AS-TEAL employs a coupled adversarial training algorithm to transfer hierarchical knowledge in taxonomies to hypernymy prediction models. We conduct extensive experiments to confirm the effectiveness of TEAL over three standard NLP tasks: unsupervised hypernymy classification, supervised hypernymy detection and graded lexical entailment. We also show that TEAL can be applied to non-English languages and can detect missing hypernymy relations in taxonomies.

IS Journal 2015 Journal Article

Knowledge Engineering with Big Data

• Xindong Wu
• Huanhuan Chen
• Gongqing Wu
• Jun Liu
• Qinghua Zheng
• Xiaofeng He
• Aoying Zhou
• Zhong-Qiu Zhao

In the era of big data, knowledge engineering faces fundamental challenges induced by fragmented knowledge from heterogeneous, autonomous sources with complex and evolving relationships. The knowledge representation, acquisition, and inference techniques developed in the 1970s and 1980s, driven by research and development of expert systems, must be updated to cope with both fragmented knowledge from multiple sources in the big data revolution and in-depth knowledge from domain experts. This article presents BigKE, a knowledge engineering framework that handles fragmented knowledge modeling and online learning from multiple information sources, nonlinear fusion on fragmented knowledge, and automated demand-driven knowledge navigation.

ICML Conference 2006 Conference Paper

R 1 -PCA: rotational invariant L 1 -norm principal component analysis for robust subspace factorization

• Chris H. Q. Ding
• Ding Zhou
• Xiaofeng He
• Hongyuan Zha

ICML Conference 2004 Conference Paper

K -means clustering via principal component analysis

• Chris H. Q. Ding
• Xiaofeng He

Principal component analysis (PCA) is a widely used statistical technique for unsupervised dimension reduction. K -means clustering is a commonly used data clustering for performing unsupervised learning tasks. Here we prove that principal components are the continuous solutions to the discrete cluster membership indicators for K -means clustering. New lower bounds for K -means objective function are derived, which is the total variance minus the eigenvalues of the data covariance matrix. These results indicate that unsupervised dimension reduction is closely related to unsupervised learning. Several implications are discussed. On dimension reduction, the result provides new insights to the observed effectiveness of PCA-based data reductions, beyond the conventional noise-reduction explanation that PCA, via singular value decomposition, provides the best low-dimensional linear approximation of the data. On learning, the result suggests effective techniques for K -means data clustering. DNA gene expression and Internet newsgroups are analyzed to illustrate our results. Experiments indicate that the new bounds are within 0.5-1.5% of the optimal values.

ICML Conference 2004 Conference Paper

Linearized cluster assignment via spectral ordering

• Chris H. Q. Ding
• Xiaofeng He

NeurIPS Conference 2001 Conference Paper

Spectral Relaxation for K-means Clustering

• Hongyuan Zha
• Xiaofeng He
• Chris Ding
• Ming Gu
• Horst Simon

The popular K-means clustering partitions a data set by minimiz(cid: 173) ing a sum-of-squares cost function. A coordinate descend method is then used to find local minima. In this paper we show that the minimization can be reformulated as a trace maximization problem associated with the Gram matrix of the data vectors. Furthermore, we show that a relaxed version of the trace maximization problem possesses global optimal solutions which can be obtained by com(cid: 173) puting a partial eigendecomposition of the Gram matrix, and the cluster assignment for each data vectors can be found by comput(cid: 173) ing a pivoted QR decomposition of the eigenvector matrix. As a by-product we also derive a lower bound for the minimum of the sum-of-squares cost function.