Jun Zhao

EAAI Journal 2026 Journal Article

A causal generative model-based optimal scheduling method for blast furnace gas system considering unknown scenarios

• Feng Jin
• Xiaoxue Wang
• Jun Zhao
• Wei Wang

Blast furnace gas is a significant category of byproduct energy source produced during the ironmaking process, and its rational utilization is crucial to improving energy efficiency in steel plants. However, frequent operator interventions continually create new operating scenarios, which makes it difficult for traditional methods to maintain effective scheduling. Existing scheduling methods based on optimization and generative adversarial networks (GANs) rely excessively on historical scenarios and fail to capture the explicit causal relationships among the key factors, thus restricting their applicability for scheduling under unknown conditions. To tackle such an issue, an optimal scheduling method for BFG system based on an improved causal generative model, which is capable of generating diverse and physically consistent scenarios, is proposed in this study. Each scenario is characterized by three interpretable factors, i. e. gas tank level, generation-consumption flow difference, and consumption of adjustable units. A causal conditional Wasserstein GAN (Causal-CWGAN) is then constructed by embedding a process-informed adjacency matrix and a differentiable acyclicity constraint into the WGAN-GP framework. In addition, a correction model combined with mechanism-based rationality rules is adopted to further optimize the consumption and filters unreasonable scenarios. Subsequently, the tank-level prediction is performed to update the generated scenario set and derive practical adjustment suggestions. Experimental results on real data from a steel enterprise show that, compared with the WGAN and the MGAN methods, the proposed one yields smaller Wasserstein distances, generates more rational scenarios, and provides adjustment strategies that can stably keep the gas tank level within the safety operating range.

AAAI Conference 2026 Conference Paper

Bias-Restrained Prefix Representation Finetuning for Mathematical Reasoning

• Sirui Liang
• Pengfei Cao
• Jian Zhao
• Cong Huang
• Jun Zhao
• Kang Liu

Parameter-Efficient finetuning (PEFT) enhances model performance on downstream tasks by updating a minimal subset of parameters. Representation finetuning (ReFT) methods further improve efficiency by freezing model weights and optimizing internal representations with fewer parameters than PEFT, outperforming PEFT on several tasks. However, ReFT exhibits a significant performance decline on mathematical reasoning tasks. To address this problem, the paper demonstrates that ReFT's poor performance on mathematical tasks primarily stems from its struggle to generate effective reasoning prefixes during the early inference phase. Moreover, ReFT disturbs the numerical encoding and the error accumulats during the CoT stage. Based on these observations, this paper proposes Bias-REstrained Prefix Representation FineTuning (BREP ReFT), which enhances ReFT's mathematical reasoning capability by truncating training data to optimize the generation of initial reasoning prefixes, intervening on the early inference stage to prevent error accumulation, and constraining the intervention vectors' magnitude to avoid disturbing numerical encoding. Extensive experiments across diverse model architectures demonstrate BREP's superior effectiveness, efficiency, and robust generalization capability, outperforming both standard ReFT and weight-based PEFT methods on the task of mathematical reasoning.

AIJ Journal 2026 Journal Article

One mind, many tongues: A deep dive into language-agnostic knowledge neurons in large language models

• Pengfei Cao
• Yuheng Chen
• Zhuoran Jin
• Yubo Chen
• Kang Liu
• Jun Zhao

Large language models (LLMs) have learned vast amounts of factual knowledge through self-supervised pre-training on large-scale corpora. Meanwhile, LLMs have also demonstrated excellent multilingual capabilities, which can express the learned knowledge in multiple languages. However, the knowledge storage mechanism in LLMs still remains mysterious. Some researchers attempt to demystify the factual knowledge in LLMs from the perspective of knowledge neurons, and subsequently discover language-agnostic knowledge neurons that store factual knowledge in a form that transcends language barriers. However, the preliminary finding suffers from two limitations: 1) High Uncertainty in Localization Results. Existing study only uses a prompt-based probe to localize knowledge neurons for each fact, while LLMs cannot provide consistent answers for semantically equivalent queries. Thus, it leads to inaccurate localization results with high uncertainty. 2) Lack of Analysis in More Languages. The study only analyzes language-agnostic knowledge neurons on English and Chinese data, without exploring more language families and languages. Naturally, it limits the generalizability of the findings. To address aforementioned problems, we first construct a new benchmark called Rephrased Multilingual LAMA (RML-LAMA), which contains high-quality cloze-style multilingual parallel queries for each fact. Then, we propose a novel method named Multilingual Integrated Gradients with Uncertainty Estimation (MATRICE), which quantifies the uncertainty across queries and languages during knowledge localization. Extensive experiments show that our method can accurately localize language-agnostic knowledge neurons. We also further investigate the role of language-agnostic knowledge neurons in cross-lingual knowledge editing, knowledge enhancement and new knowledge injection.

AAAI Conference 2026 Conference Paper

Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination

• Mingqi Wu
• Zhihao Zhang
• Qiaole Dong
• Zhiheng Xi
• Jun Zhao
• Senjie Jin
• Xiaoran Fan
• Yuhao Zhou

Reasoning in large language models has long been a central research focus, and recent studies employing reinforcement learning (RL) have introduced diverse methods that yield substantial performance gains with minimal or even no external supervision. Surprisingly, some studies even suggest that random or incorrect reward signals can enhance performance. However, these breakthroughs are predominantly observed for the mathematically strong Qwen2.5 series on benchmarks such as MATH-500, AMC, and AIME, and seldom transfer to models like Llama, which warrants a more in-depth investigation. In this work, our empirical analysis reveals that pre-training on massive web-scale corpora leaves Qwen2.5 susceptible to data contamination in widely used benchmarks. Consequently, conclusions derived from contaminated benchmarks on Qwen2.5 series may be unreliable. To obtain trustworthy evaluation results, we introduce a generator that creates fully clean arithmetic problems of arbitrary length and difficulty, dubbed RandomCalculation. Using this leakage-free dataset, we show that only accurate reward signals yield steady improvements that surpass the base model’s performance boundary in mathematical reasoning, whereas random or incorrect rewards do not. Moreover, we conduct more fine-grained analyses to elucidate the factors underlying the different performance observed on the MATH-500 and RandomCalculation benchmarks. Consequently, we recommend that future studies evaluate models on uncontaminated benchmarks and, when feasible, test various model series to ensure trustworthy conclusions about RL and related methods.

AAAI Conference 2026 Conference Paper

RetroLM: Retrieval-Augmented KVs for Long-Context Processing

• Kun Luo
• Zheng Liu
• Shitao Xiao
• Jiabei Chen
• Hongjin Qian
• Peitian Zhang
• Shanshan Jiang
• Bin Dong

Long-context processing remains a significant challenge for large language models (LLMs). Retrieval-augmented generation (RAG) has recently emerged as a promising approach, enabling LLMs to selectively access relevant information from extended contexts to improve efficiency. However, existing RAG approaches often lag behind other efficient long-context processing methods primarily due to inherent limitations on inaccurate retrieval and fragmented contexts. To address these limitations, we propose RetroLM, a novel RAG framework designed for effective long-context processing. Unlike traditional approaches, RetroLM introduces KV-level retrieval augmentation, which partitions the LLM's KV cache into contiguous pages and performs encoding and decoding operations based on the retrieved KV pages. Built upon this framework, we further develop a specialized retriever for precise retrieval of critical pages and conduct unsupervised post-training to optimize the model’s ability to leverage retrieved information. Compared with traditional RAG, the new approach enhances robustness to retrieval inaccuracy, facilitates effective utilization of fragmented contexts, and saves the cost from repeated context-encoding operations. We conduct extensive evaluations across several popular benchmarks, including LongBench, InfiniteBench, and RULER. RetroLM consistently outperforms existing long-LLMs and RAG-based methods, especially in tasks requiring deep reasoning or extreme context lengths.

AAAI Conference 2026 Conference Paper

SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning

• Huanxuan Liao
• Yixing Xu
• Shizhu He
• Guanchen Li
• Xuanwu Yin
• Dong Li
• Emad Barsoum
• Jun Zhao

Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention computation scales quadratically. Existing approaches address this issue by compressing the KV cache along the temporal axis through strategies such as token eviction or merging to reduce memory and computational overhead. However, these methods often neglect fine-grained importance variations across feature dimensions (i.e., the channel axis), thereby limiting their ability to effectively balance efficiency and model accuracy. In reality, we observe that channel saliency varies dramatically across both queries and positions: certain feature channels carry near-zero information for a given query, while others spike in relevance. To address this oversight, we propose SPARK, a training-free plug-and-play method that applies unstructured sparsity by pruning KV at the channel level, while dynamically restoring the pruned entries during attention score computation. Notably, our approach is orthogonal to existing KV compression and quantization techniques, making it compatible for integration with them to achieve further acceleration. By reducing channel-level redundancy, SPARK enables processing of longer sequences within the same memory budget. For sequences of equal length, SPARK not only preserves or improves model accuracy but also reduces KV cache storage by over 30% compared to eviction-based methods. Furthermore, even in an aggressive pruning ratio of 80%, SPARK maintains performance with less degradation than 5% compared to the based eviction method, demonstrating robustness and effectiveness. Our code will be available at \url{https://github.com/AMD-AIG-AIMA/AMD-Spark}.

AAAI Conference 2026 Conference Paper

TaREx: Reinforcement Learning for Code-Driven Table Reasoning

• Fangyu Lei
• Jinxiang Meng
• Yiming Huang
• Shizhu He
• Jun Zhao
• Kang Liu

Automatically solving table reasoning tasks remains challenging due to three main factors: (1) diverse and hierarchical table structures that hinder comprehension, (2) the heavy reliance on complex logical and numerical reasoning—which makes purely text-based methods prone to hallucinations—and (3) the necessity of multi-step processing to handle intricate tasks involving multiple and lengthy tables. To address these challenges, we introduce TaREx, a novel framework that unifies table representation, integrates code-driven execution, and supports interactive multi-step reasoning. TaREx employs a reinforcement learning-based training pipeline to optimize its reasoning policy for complex tasks. Experimental results show that TaREx achieves state-of-the-art performance across a wide range of table reasoning benchmarks, both in-domain and out-of-domain. These include fundamental tasks such as table question answering (TQA) and table fact verification (TFV), as well as advanced tabular data analysis tasks. The results highlight TaREx’s effectiveness and scalability in advancing automated table reasoning.

EAAI Journal 2025 Journal Article

A novel framework for assessing determinant risk factors on cyber (dis)trust behaviors of netizens in deepfakes

• Milad Taleby Ahvanooey
• Wojciech Mazurczyk
• Zefan Wang
• Jun Zhao

Nowadays, Generative Artificial Intelligence (GenAI) tools or trainable agents can craft synthetic media (hereafter referred to as deepfakes) in the form of realistic texts, images, videos, and audios, incorporating events or things that never occurred in real life. These GenAI tools empower marketers and malicious actors to create deepfakes, both authorized and weaponized multimedia, which allows them to include celebrities without appearing in front of cameras or creating seductive phishing scams. Although GenAI tools can reduce the cost of content construction, they enable new risky opportunities (e. g. , deepfake phishing and cyberbullying) that negatively impact netizens’ learning and (dis)trust behaviors in cyberspace. To address such risks, this study proposes a Multi-Criteria-Multi-Decision-Makers (MCMDM)-based Deepfake Risk Assessment Framework (DeepFakeR-MF) to evaluate determinant factors that impact the cyber (dis)trust behaviors of netizens in deepfakes. Moreover, DeepFakeR-MF deploys a combination of a novel optimized spherical fuzzy analytic hierarchy process method and a game theory-based MCMDM approach to prioritize and recommend alternative strategies that can be taken by five management sectors (e. g. , industrial enterprises, governmental organizations, media outlets, social non-profit, and educational institutes) to mitigate GenAI-associated risks. Then, we collect 100 experts’ judgments by analyzing their responses to our questionnaire and prioritize the importance of determinant factors considering their preferences. To validate the prioritized factors on the performance of DeepFakeR-MF, we conduct a sensitivity analysis applying Monte Carlo statistical modeling. Finally, our results confirm that DeepFakeR-MF provides effective strategic alternatives for policymakers, educators, media professionals, engineers, and netizens, hopefully reducing the socio-economic risks of deepfakes.

AAAI Conference 2025 Conference Paper

CITI: Enhancing Tool Utilizing Ability in Large Language Models Without Sacrificing General Performance

• Yupu Hao
• Pengfei Cao
• Zhuoran Jin
• Huanxuan Liao
• Yubo Chen
• Kang Liu
• Jun Zhao

Tool learning enables Large Language Models (LLMs) to interact with the external environment by invoking tools, enriching the accuracy and capability scope of LLMs. However, previous works predominantly focus on improving the model's tool-utilizing accuracy and the ability to generalize to new, unseen tools, excessively forcing LLMs to adjust specific tool-invoking pattern without considering the harm to the model's general performance. This deviates from the actual applications and original intention of integrating tools to enhance the model. To tackle this problem, we dissect the capability trade-offs by examining the hidden representation changes and the gradient-based importance score of the model's components. Based on the analysis result, we propose a Component Importance-based Tool-utilizing ability Injection method (CITI). According to the gradient-based importance score of different components, it alleviates the capability conflicts caused by the fine-tuning process by applying distinct training strategies to different components. CITI applies Mixture-Of-LoRA (MOLoRA) for important components. Meanwhile, it fine-tunes the parameters of a few components deemed less important in the backbone of the LLM, while keeping other parameters frozen. CITI can effectively enhance the model's tool-utilizing capability without excessively compromising its general performance. Experimental results demonstrate that our approach achieves outstanding performance across a range of evaluation metrics.

ICML Conference 2025 Conference Paper

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling

• Xiang Hu
• Zhihao Teng
• Jun Zhao
• Wei Wu
• Kewei Tu

Despite the success of Transformers, handling longer contexts remains challenging due to the limited length generalization and quadratic complexity of self-attention, which often requires post-training with a larger attention window, significantly increasing computational and memory costs. In this paper, we propose a novel attention mechanism based on dynamic context, Grouped Cross Attention (GCA), which can generalize to 1000 $\times$ the pre-training context length while maintaining the ability to access distant information with a constant attention window size. For a given input sequence, we split it into chunks and use each chunk to retrieve top-$k$ relevant past chunks for subsequent text generation. Specifically, unlike most previous works that use an off-the-shelf retriever, our key innovation allows the retriever to learn how to retrieve past chunks that better minimize the auto-regressive loss of subsequent tokens in an end-to-end manner, which adapts better to causal language models. Such a mechanism accommodates retrieved chunks with a fixed-size attention window to achieve long-range information access, significantly reducing computational and memory costs during training and inference. Experiments show that GCA-based models achieve near-perfect accuracy in passkey retrieval for 16M context lengths, which is $1000 \times$ the training length.

EAAI Journal 2025 Journal Article

Frequency domain nuances guided parallel transformer model for industrial anomaly localization

• Jun Zhao
• Kaixuan Yu
• Yu Miao
• Yingsen Wang
• Yue Ma
• Jiawei Zhang
• Juanjuan Zhao
• Yan Qiang

Unsupervised visual anomaly localization research has garnered significant attention in industrial component surface quality inspection tasks, particularly in realistic scenarios characterized by extreme imbalance between positive and negative samples. While several existing approaches endeavor to design hybrid architectures of convolutional neural networks (CNNs) and vision transformers (ViTs) to enhance the performance capability of unsupervised models, their overall effectiveness remains unstable. In this paper, we introduce a novel approach known as frequency domain nuances mining (FDNM), presenting a parallel transformer framework for anomaly localization. This framework leverages the inductive bias property of CNNs for local spatial understanding and the advantageous ViTs for global representation learning, enabling simultaneous capture of local correlations and global information. To further enhance the model’s discriminative ability for subtle differences, we propose a frequency domain decoupling mechanism that exploits the frequency domain properties of images to improve model interpretability. Specifically, multi-granularity differences among frequency domain components serve as prior embeddings in FDNM, augmenting the potential feature representation of local details and global semantics. Furthermore, we synergistically train frequency domain invariant sampling loss and focal loss to balance the differential representation of features in the cross-frequency domain, leading to a more stable training process and more accurate anomaly localization results. Experiments conducted on the challenging 15-class industrial anomaly detection dataset validate the superiority of the proposed method. The image-level area under the receiver operating characteristic curve (AUROC), pixel-level AUROC and average precision (AP) scores of FDNM reach 99. 0%, 98. 7% and 68. 7%, respectively, demonstrating comparable performance to state-of-the-art methods.

NeurIPS Conference 2025 Conference Paper

Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access

• Xiang Hu
• Jiaqi Leng
• Jun Zhao
• Kewei Tu
• Wei Wu

A key advantage of Recurrent Neural Networks (RNNs) over Transformers is their linear computational and space complexity enables faster training and inference for long sequences. However, RNNs are fundamentally unable to randomly access historical context, and simply integrating attention mechanisms may undermine their efficiency advantages. To overcome this limitation, we propose \textbf{H}ierarchical \textbf{S}parse \textbf{A}ttention (HSA), a novel attention mechanism that enhances RNNs with long-range random access flexibility while preserving their merits in efficiency and length generalization. HSA divides inputs into chunks, selecting the top-$k$ chunks and hierarchically aggregates information. The core innovation lies in learning token-to-chunk relevance based on fine-grained token-level information inside each chunk. This approach enhances the precision of chunk selection across both in-domain and out-of-domain context lengths. To make HSA efficient, we further introduce a hardware-aligned kernel design. By combining HSA with Mamba, we introduce RAMba, which achieves perfect accuracy in passkey retrieval across 64 million contexts despite pre-training on only 4K-length contexts, and significant improvements on various downstream tasks, with nearly constant memory footprint. These results show RAMba's huge potential in long-context modeling.

AAAI Conference 2025 Conference Paper

Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models

• Chenhui Hu
• Pengfei Cao
• Yubo Chen
• Kang Liu
• Jun Zhao

Knowledge editing aims to update outdated or incorrect knowledge in large language models (LLMs). However, current knowledge editing methods have limited scalability for lifelong editing. This study explores the fundamental reason why knowledge editing fails in lifelong editing. We begin with the closed-form solution derived from linear associative memory, which underpins state-of-the-art knowledge editing methods. We extend the solution from single editing to lifelong editing, and through rigorous mathematical derivation, identify an interference term in the final solution, suggesting that editing knowledge may impact irrelevant knowledge. Further analysis of the interference term reveals a close relationship with superposition between knowledge representations. When knowledge superposition does not exist in language models, the interference term vanishes, allowing for lossless knowledge editing. Experiments across numerous language models reveal that knowledge superposition is universal, exhibiting high kurtosis, zero mean, and heavy-tailed distributions with clear scaling laws. Ultimately, by combining theory and experiments, we demonstrate that knowledge superposition is the fundamental reason for the failure of lifelong editing. Moreover, this is the first study to investigate knowledge editing from the perspective of superposition and provides a comprehensive observation of superposition across numerous real-world language models.

EAAI Journal 2025 Journal Article

Knowledge-based real-time scheduling for gas supply network using cooperative multi-agent reinforcement learning and predictive functional range control

• Pengwei Zhou
• Zuhua Xu
• Jiakun Fang
• Jun Zhao
• Chunyue Song
• Zhijiang Shao

In steel enterprises, the real-time scheduling optimization of the gas supply network can provide strong support for stabilizing production and enhancing economic benefits. Due to the coupling of multiple gas/liquid products and numerous units, centralized scheduling methods require large training and coordination overhead. Consequently, the real-time scheduling problem of a multi-product gas supply network is modeled under the cooperative multi-agent reinforcement learning (MARL) architecture, which makes the scheduling strategy of each unit keep the same optimization goal. Decentralized execution mode reduces the computing cost and information exchange compared with centralized execution mode. Different from adding constraint penalty terms as soft constraints, a constraint monitor module is designed by utilizing the process knowledge, ensuring the various production constraints are satisfied. This strategy can reduce trail-and-error costs, making it more conducive to industrial safety. To deal with the unknown disturbance in production (typically gas leakage), a predictive functional range control (PFRC) algorithm is then developed to modify the future gas demand. Finally, case studies are carried out on a real-world gas supply network to verify the performance of the proposed method.

TIST Journal 2025 Journal Article

MGRL4RE: A Multi-Graph Representation Learning Approach for Urban Region Embedding

• Meng Chen
• Zechen Li
• Hongwei Jia
• Xin Shao
• Jun Zhao
• Qiang Gao
• Min Yang
• Yilong Yin

Using multi-modal data to learn region representations has gained popularity for its ability to reveal diverse socioeconomic features in cities. However, many studies focus solely on semantic features from points-of-interest (POIs), neglecting the issue of spatial imbalance. This article introduces a Multi-Graph Representation Learning framework for Region Embedding (MGRL4RE), which leverages both inter-region and intra-region correlations through two main components: multi-graph construction based on various region correlations and multi-graph representation learning. The construction module creates a multi-graph reflecting various correlations among regions, utilizing geo-tagged POIs, region data, and human mobility data. Specifically, we assess a region’s importance relative to its spatial context (neighborhood) and develop spatially invariant semantic features to address spatial imbalance. Furthermore, the representation learning module generates comprehensive and effective region representations via multi-view embedding fusion. Our extensive experiments across various downstream tasks, including land use clustering, region popularity prediction, and crime prediction, confirm that our model significantly outperforms existing state-of-the-art region embedding methods.

AAAI Conference 2025 Conference Paper

Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks

• Huanxuan Liao
• Shizhu He
• Yao Xu
• Yuanzhe Zhang
• Kang Liu
• Jun Zhao

In this paper, we propose Neural-Symbolic Collaborative Distillation (NesyCD), a novel knowledge distillation method for learning the complex reasoning abilities of Large Language Models (LLMs, e.g., \textgreater 13B). We argue that complex reasoning tasks are difficult for Small Language Models (SLMs, e.g., $\leq$ 7B), as these tasks demand not only general cognitive abilities but also specialized knowledge, which is often sparse and difficult for these neural-based SLMs to effectively capture. Therefore, NesyCD distills the general capabilities and specialized knowledge in LLMs using different manners.On the one hand, we distill only general abilities from teacher LLMs into the student SLMs of parameterized neural networks. On the other hand, for the specialized abilities and uncommon knowledge of a complex reasoning task, we employ a symbolic knowledge distillation approach to obtain and store the specialized knowledge within a symbolic knowledge base (KB).By decoupling general and specialized capabilities, the proposed NesyCD can achieve superior performance cost-effectively, utilizing smaller models and blending parameterized neural networks with symbolic KB. Moreover, the specialized KB generalizes well and is comprehended and manipulated by humans.Our experiments show that NesyCD significantly boosts SLMs' complex reasoning performance on in-domain (BBH, GSM8K) and out-of-domain (AGIEval, ARC) datasets. Notably, our approach enabled the LLaMA3-8B and Qwen2-7B to surpass GPT-3.5-turbo in performance and come close to matching LLaMA3-70B, despite the latter having nine times more parameters.

NeurIPS Conference 2025 Conference Paper

RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality

• Chenlong Zhang
• Zhuoran Jin
• Hongbang Yuan
• Jiaheng Wei
• Tong Zhou
• Kang Liu
• Jun Zhao
• Yubo Chen

The widespread deployment of Large Language Models (LLMs) trained on massive, uncurated corpora has raised growing concerns about the inclusion of sensitive, copyrighted, or illegal content. This has led to increasing interest in LLM unlearning: the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility. However, existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss. In this work, we propose $\textbf{R}$einforcement $\textbf{U}$n$\textbf{LE}$arning ($\textbf{RULE}$), an efficient framework that formulates unlearning as a refusal boundary optimization problem. RULE is trained with a small portion of forget set and synthesized boundary queries, using a verifiable reward function that encourages safe refusal on forget-related queries while preserving helpful responses on permissible inputs. We provide both theoretical and empirical evidence demonstrating the effectiveness of RULE in achieving targeted unlearning without compromising model utility. Experimental results show that, with only 12\% forget set and 8\% synthesized boundary data, RULE outperforms existing baselines by up to $17. 4\%$ forget quality and $16. 3\%$ naturalness response while maintaining general utility, achieving $\textit{forget-retain Pareto Optimality}$. Remarkably, we further observe that RULE improves the $\textit{naturalness}$ of model outputs, enhances training $\textit{efficiency}$, and exhibits strong $\textit{generalization ability}$, generalizing refusal behavior to semantically related but unseen queries.

AAAI Conference 2025 Conference Paper

Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models

• Hongbang Yuan
• Zhuoran Jin
• Pengfei Cao
• Yubo Chen
• Kang Liu
• Jun Zhao

LLM have achieved success in many fields but still troubled by problematic content in the training corpora. LLM unlearning aims at reducing their influence and avoid undesirable behaviours. However, existing unlearning methods remain vulnerable to adversarial queries and the unlearned knowledge resurfaces after the manually designed attack queries. As part of a red-team effort to proactively assess the vulnerabilities of unlearned models, we design Dynamic Unlearning Attack (DUA), a dynamic and automated framework to attack these models and evaluate their robustness. It optimizes adversarial suffixes to reintroduce the unlearned knowledge in various scenarios. We find that unlearned knowledge can be recovered in 55.2% of the questions, even without revealing the unlearned model's parameters. In response to this vulnerability, we propose Latent Adversarial Unlearning (LAU), a universal framework that effectively enhances the robustness of the unlearned process. It formulates the unlearning process as a min-max optimization problem and resolves it through two stages: an attack stage, where perturbation vectors are trained and added to the latent space of LLMs to recover the unlearned knowledge, and a defense stage, where previously trained perturbation vectors are used to enhance unlearned model's robustness. With our LAU framework, we obtain two robust unlearning methods, AdvGA and AdvNPO. We conduct extensive experiments across multiple unlearning benchmarks and various models, and demonstrate that they improve the unlearning effectiveness by over 53.5%, cause only less than a 11.6% reduction in neighboring knowledge, and have almost no impact on the model's general capabilities.

NeurIPS Conference 2025 Conference Paper

Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models

• Jun Zhao
• Yongzhuo Yang
• Xiang Hu
• Jingqi Tong
• Yi Lu
• Wei Wu
• Tao Gui
• Qi Zhang

Retrieval-Augmented Generation (RAG) provides additional contextual knowledge to complement the parametric knowledge in Large Language Models (LLMs). These two knowledge interweave to enhance the accuracy and timeliness of LLM responses. However, the internal mechanisms by which LLMs utilize these knowledge remain unclear. We propose modeling the forward propagation of knowledge as an entity flow, employing this framework to trace LLMs' internal behaviors when processing mixed-source knowledge. Linear probing utilizes a trainable linear classifier to detect specific attributes in hidden layers. However, once trained, a probe cannot adapt to dynamically specified entities. To address this challenge, we construct an entity-aware probe, which introduces special tokens to mark probing targets and employs a small trainable rank-8 lora update to process these special markers. We first verify this approach through an attribution experiment, demonstrating that it can accurately detect information about ad-hoc entities from complex hidden states. Next, we trace entity flows across layers to understand how LLMs reconcile conflicting knowledge internally. Our probing results reveal that contextual and parametric knowledge are routed between tokens through distinct sets of attention heads, supporting attention competition only within knowledge types. While conflicting knowledge maintains a residual presence across layers, aligned knowledge from multiple sources gradually accumulates, with the magnitude of this accumulation directly determining its influence on final outputs.

ICLR Conference 2025 Conference Paper

Verifying Properties of Binary Neural Networks Using Sparse Polynomial Optimization

• Jianting Yang
• Srecko Ðurasinovic
• Jean-Bernard Lasserre
• Victor Magron
• Jun Zhao

This paper explores methods for verifying the properties of Binary Neural Networks (BNNs), focusing on robustness against adversarial attacks. Despite their lower computational and memory needs, BNNs, like their full-precision counterparts, are also sensitive to input perturbations. Established methods for solving this problem are predominantly based on Satisfiability Modulo Theories and Mixed-Integer Linear Programming techniques, which are characterized by NP complexity and often face scalability issues. We introduce an alternative approach using Semidefinite Programming relaxations derived from sparse Polynomial Optimization. Our approach, compatible with continuous input space, not only mitigates numerical issues associated with floating-point calculations but also enhances verification scalability through the strategic use of tighter first-order semidefinite relaxations. We demonstrate the effectiveness of our method in verifying robustness against both $\||.|\|_\infty$ and $\||.|\|_2$-based adversarial attacks.

AAAI Conference 2024 Conference Paper

A Joint Framework with Heterogeneous-Relation-Aware Graph and Multi-Channel Label Enhancing Strategy for Event Causality Extraction

• Ruili Pu
• Yang Li
• Jun Zhao
• Suge Wang
• Deyu Li
• Jian Liao
• Jianxing Zheng

Event Causality Extraction (ECE) aims to extract the cause-effect event pairs with their structured event information from plain texts. As far as we know, the existing ECE methods mainly focus on the correlation between arguments, without explicitly modeling the causal relationship between events, and usually design two independent frameworks to extract cause events and effect events, respectively, which cannot effectively capture the dependency between the subtasks. Therefore, we propose a joint multi-label extraction framework for ECE to alleviate the above limitations. In particular, 1) we design a heterogeneous-relation-aware graph module to learn the potential relationships between events and arguments, in which we construct the heterogeneous graph by taking the predefined event types and all the words in the sentence as nodes, and modeling three relationships of "event-event", "event-argument" and "argument-argument" as edges. 2) We also design a multi-channel label enhancing module to better learn the distributed representation of each label in the multi-label extraction framework, and further enhance the interaction between the subtasks by considering the preliminary results of cause-effect type identification and event argument extraction. The experimental results on the benchmark dataset ECE-CCKS show that our approach outperforms previous state-of-the-art methods, and that our model also performs well on the complex samples with multiple cause-effect event pairs.

NeurIPS Conference 2024 Conference Paper

From Instance Training to Instruction Learning: Task Adapters Generation from Instructions

• Huanxuan Liao
• Shizhu He
• Yao Xu
• Yuanzhe Zhang
• Yanchao Hao
• Shengping Liu
• Kang Liu
• Jun Zhao

Large language models (LLMs) have acquired the ability to solve general tasks by utilizing instruction finetuning (IFT). However, IFT still relies heavily on instance training of extensive task data, which greatly limits the adaptability of LLMs to real-world scenarios where labeled task instances are scarce and broader task generalization becomes paramount. Contrary to LLMs, humans acquire skills and complete tasks not merely through repeated practice but also by understanding and following instructional guidelines. This paper is dedicated to simulating human learning to address the shortcomings of instance training, focusing on instruction learning to enhance cross-task generalization. Within this context, we introduce Task Adapters Generation from Instructions (TAGI), which automatically constructs the task-specific model in a parameter generation manner based on the given task instructions without retraining for unseen tasks. Specifically, we utilize knowledge distillation to enhance the consistency between TAGI developed through Learning with Instruction and task-specific models developed through Training with Instance, by aligning the labels, output logits, and adapter parameters between them. TAGI is endowed with cross-task generalization capabilities through a two-stage training process that includes hypernetwork pretraining and finetuning. We evaluate TAGI on the Super-Natural Instructions and P3 datasets. The experimental results demonstrate that TAGI can match or even outperform traditional meta-trained models and other hypernetwork models, while significantly reducing computational requirements. Our code will be available at https: //github. com/Xnhyacinth/TAGI.

AAAI Conference 2024 Conference Paper

Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons

• Yuheng Chen
• Pengfei Cao
• Yubo Chen
• Kang Liu
• Jun Zhao

Pre-trained language models (PLMs) contain vast amounts of factual knowledge, but how the knowledge is stored in the parameters remains unclear. This paper delves into the complex task of understanding how factual knowledge is stored in multilingual PLMs, and introduces the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely compared to current methods, and is more universal across various architectures and languages. Moreover, we conduct an in-depth exploration on knowledge neurons, leading to the following two important discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which store factual knowledge in a form that transcends language. We design cross-lingual knowledge editing experiments, demonstrating that the PLMs can accomplish this task based on language-independent neurons; (2) The discovery of Degenerate Knowledge Neurons, a novel type of neuron showing that different knowledge neurons can store the same fact. Its property of functional overlap endows the PLMs with a robust mastery of factual knowledge. We design fact-checking experiments, proving that the degenerate knowledge neurons can help the PLMs to detect wrong facts. Experiments corroborate these findings, shedding light on the mechanisms of factual knowledge storage in multilingual PLMs, and contribute valuable insights to the field. The code is available at https://github.com/heng840/AMIG.

IJCAI Conference 2024 Conference Paper

Oasis: Data Curation and Assessment System for Pretraining of Large Language Models

• Tong Zhou
• Yubo Chen
• Pengfei Cao
• Kang Liu
• Shengping Liu
• Jun Zhao

Data is one of the most critical elements in building a large language model. However, existing systems either fail to customize a corpus curation pipeline or neglect to leverage comprehensive corpus assessment for iterative optimization of the curation. To this end, we present a pretraining corpus curation and assessment platform called Oasis — a one-stop system for data quality improvement and quantification with user-friendly interactive interfaces. Specifically, the interactive modular rule filter module can devise customized rules according to explicit feedback. The debiased neural filter module builds the quality classification dataset in a negative-centric manner to remove the undesired bias. The adaptive document deduplication module could execute large-scale deduplication with limited memory resources. These three parts constitute the customized data curation module. And in the holistic data assessment module, a corpus can be assessed in local and global views, with three evaluation means including human, GPT-4, and heuristic metrics. We exhibit a complete process to use Oasis for the curation and assessment of pretraining data. In addition, an 800GB bilingual corpus curated by Oasis is publicly released.

NeurIPS Conference 2024 Conference Paper

RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models

• Zhuoran Jin
• Pengfei Cao
• Chenhao Wang
• Zhitao He
• Hongbang Yuan
• Jiachun Li
• Yubo Chen
• Kang Liu

Large language models (LLMs) inevitably memorize sensitive, copyrighted, and harmful knowledge from the training corpus; therefore, it is crucial to erase this knowledge from the models. Machine unlearning is a promising solution for efficiently removing specific knowledge by post hoc modifying models. In this paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM unlearning. RWKU is designed based on the following three key factors: (1) For the task setting, we consider a more practical and challenging unlearning setting, where neither the forget corpus nor the retain corpus is accessible. (2) For the knowledge source, we choose 200 real-world famous people as the unlearning targets and show that such popular knowledge is widely present in various LLMs. (3) For the evaluation framework, we design the forget set and the retain set to evaluate the model’s capabilities across various real-world applications. Regarding the forget set, we provide four four membership inference attack (MIA) methods and nine kinds of adversarial attack probes to rigorously test unlearning efficacy. Regarding the retain set, we assess locality and utility in terms of neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. We conduct extensive experiments across two unlearning scenarios, two models and six baseline methods and obtain some meaningful findings. We release our benchmark and code publicly at http: //rwku-bench. github. io for future work.

TIST Journal 2023 Journal Article

A Spatial and Adversarial Representation Learning Approach for Land Use Classification with POIs

• Ronghui Xu
• Weiming Huang
• Jun Zhao
• Meng Chen
• Liqiang Nie

Points-of-interests (POIs) have been proven to be indicative for sensing urban land use in numerous studies. However, recent progress mainly relies on spatial co-occurrence patterns among POI categories, which falls short in utilizing the rich semantic information embodied in POI hierarchical categories and in sensing the spatial distribution patterns of POIs at an individual zonal scale. In this context, we present a spatial and adversarial representation learning approach (SARL) for predicting land use of urban zones with POIs. SARL deeply mines the information from POIs from both spatial and categorical perspectives. Specifically, we first utilize a convolutional neural network to sense the spatial distribution patterns of POIs in each urban zone. We then leverage an autoencoder and an adversarial learning strategy to mine the POI categorical information in all hierarchical levels, which emphasizes the prominent and definitive POIs while preserves the overall POI hierarchical structures in each zone. Finally, we fuse these information from the two perspectives via a Wide & Deep network and carry out land use prediction with the fused embeddings. We conduct comprehensive experiments to validate the effectiveness of SARL in four European cities with real-world data. The results demonstrate that SARL substantially outperforms several competitive baselines.

TCS Journal 2023 Journal Article

Bounded-collusion decentralized ABE with sublinear parameters

• Jun Zhao
• Minghao Yang
• Junqing Gong
• Kai Zhang
• Haifeng Qian

In this paper, we propose a decentralized ABE scheme against bounded collusion which means the number of users in the system is a-prior bounded. The scheme enjoys public key and ciphertext of sublinear sizes in the number of users in the system while all prior constructions require linear sizes. Besides, our scheme achieves semi-adaptive security under bilateral k-Lin assumption and SXDH assumption in a pairing group. Keep the same as the previous constructions, the scheme supports monotone span program as a policy and does not rely on the random oracle. Technically, we follow Wang et al. 's “linear secret sharing scheme (LSSS) + inner-product functional encryption (IPFE)” paradigm [PKC'19] and use (an extended variant of) functional encryption for quadratic functions (QFE) in the place of IPFE. By this, we encrypt with sublinear-size random coins and later expand them to linear-size entropy for security proof. Roughly, the use of QFE requires bilateral k-Lin assumption while the entropy expansion relies on SXDH.

EAAI Journal 2023 Journal Article

Deep Gaussian mixture adaptive network for robust soft sensor modeling with a closed-loop calibration mechanism

• Xiangrui Zhang
• Chunyue Song
• Jun Zhao
• Zuhua Xu

Process drift leads to an out-of-distribution problem between historical training data and online deployment data, which deteriorates the performance of soft sensors. Unfortunately, few existing deep learning soft sensors have calibration mechanisms to deal with the process drift. To this end, this article proposes a robust Deep Gaussian Mixture Adaptive Network (DGMAN)-based soft sensor with a closed-loop calibration mechanism. First, a Gaussian Mixture Conditional Variational Autoencoder (GMCVAE) is presented, with the ability to capture the multimodality in industrial data. Afterwards, we propose a novel semi-supervised Gaussian mixture domain adaptation to perform conditional and marginal probability distribution alignment in the Gaussian mixture probabilistic latent space created by GMCVAE, using both labeled and unlabeled target samples. Theoretically, two kinds of process drift, including concept drift and virtual drift, could be alleviated by conditional distribution adaptation and marginal distribution adaptation, respectively. In addition, a new closed-loop calibration mechanism is deployed in a pre-training and fine-tuning fashion. Finally, robust analysis in theory and experimental results on the gas turbine dataset demonstrate that the proposed DGMAN-based soft sensor with robustness against process drift can maintain long-term validity in real industrial processes. Calibrated by only 100 labeled samples per year, the RMSE of DGMAN is 5. 55 ± 0. 32 after three years of deployment in the power plant, outperforming all comparison methods.

AAAI Conference 2023 Conference Paper

Event Process Typing via Hierarchical Optimal Transport

• Bo Zhou
• Yubo Chen
• Kang Liu
• Jun Zhao

Understanding intention behind event processes in texts is important to many applications. One challenging task in this line is event process typing, which aims to tag the process with one action label and one object label describing the overall action of the process and object the process likely affects respectively. To tackle this task, existing methods mainly rely on the matching of the event process level and label level representation, which ignores two important characteristics: Process Hierarchy and Label Hierarchy. In this paper, we propose a Hierarchical Optimal Transport (HOT) method to address the above problem. Specifically, we first explicitly extract the process hierarchy and label hierarchy. Then the HOT optimally matches the two types of hierarchy. Experimental results show that our model outperforms the baseline models, illustrating the effectiveness of our model.

AAAI Conference 2023 Conference Paper

Zero-Shot Cross-Lingual Event Argument Extraction with Language-Oriented Prefix-Tuning

• Pengfei Cao
• Zhuoran Jin
• Yubo Chen
• Kang Liu
• Jun Zhao

Event argument extraction (EAE) aims to identify the arguments of a given event, and classify the roles that those arguments play. Due to high data demands of training EAE models, zero-shot cross-lingual EAE has attracted increasing attention, as it greatly reduces human annotation effort. Some prior works indicate that generation-based methods have achieved promising performance for monolingual EAE. However, when applying existing generation-based methods to zero-shot cross-lingual EAE, we find two critical challenges, including Language Discrepancy and Template Construction. In this paper, we propose a novel method termed as Language-oriented Prefix-tuning Network (LAPIN) to address the above challenges. Specifically, we devise a Language-oriented Prefix Generator module to handle the discrepancies between source and target languages. Moreover, we leverage a Language-agnostic Template Constructor module to design templates that can be adapted to any language. Extensive experiments demonstrate that our proposed method achieves the best performance, outperforming the previous state-of-the-art model by 4.8% and 2.3% of the average F1-score on two multilingual EAE datasets.

AAAI Conference 2021 System Paper

CogNet: Bridging Linguistic Knowledge, World Knowledge and Commonsense Knowledge

• Chenhao Wang
• Yubo Chen
• Zhipeng Xue
• Yang Zhou
• Jun Zhao

In this paper, we present CogNet, a knowledge base (KB) dedicated to integrating three types of knowledge: (1) linguistic knowledge from FrameNet, which schematically describes situations, objects and events. (2) world knowledge from YAGO, Freebase, DBpedia and Wikidata, which provides explicit knowledge about specific instances. (3) commonsense knowledge from ConceptNet, which describes implicit general facts. To model these different types of knowledge consistently, we introduce a three-level unified frame-styled representation architecture. To integrate free-form commonsense knowledge with other structured knowledge, we propose a strategy that combines automated labeling and crowdsourced annotation. At present, CogNet integrates 1, 000+ semantic frames from linguistic KBs, 20, 000, 000+ frame instances from world KBs, as well as 90, 000+ commonsense assertions from commonsense KBs. All these data can be easily queried and explored on our online platform, and free to download in RDF format for utilization under a CC-BY-SA 4. 0 license. The demo and data are available at http: //cognet. top/.

AAAI Conference 2021 Short Paper

Generating Adversarial yet Inconspicuous Patches with a Single Image (Student Abstract)

• Jinqi Luo
• Tao Bai
• Jun Zhao

Deep neural networks have been shown vulnerable to adversarial patches, where exotic patterns can result in model’s wrong prediction. Nevertheless, existing approaches to adversarial patch generation hardly consider the contextual consistency between patches and the image background, causing such patches to be easily detected by human observation. Additionally, these methods require a large amount of data for training, which is computationally expensive. To overcome these challenges, we propose an approach to generate adversarial yet inconspicuous patches with one single image. In our approach, adversarial patches are produced in a coarse-to-fine way with multiple scales of generators and discriminators. The selection of patch location is based on the perceptual sensitivity of victim models. Contextual information is encoded during the Min-Max training to make patches consistent with surroundings.

IJCAI Conference 2021 Conference Paper

MG-DVD: A Real-time Framework for Malware Variant Detection Based on Dynamic Heterogeneous Graph Learning

• Chen Liu
• Bo Li
• Jun Zhao
• Ming Su
• Xu-Dong Liu

Detecting the newly emerging malware variants in real time is crucial for mitigating cyber risks and proactively blocking intrusions. In this paper, we propose MG-DVD, a novel detection framework based on dynamic heterogeneous graph learning, to detect malware variants in real time. Particularly, MG-DVD first models the fine-grained execution event streams of malware variants into dynamic heterogeneous graphs and investigates real-world meta-graphs between malware objects, which can effectively characterize more discriminative malicious evolutionary patterns between malware and their variants. Then, MG-DVD presents two dynamic walk-based heterogeneous graph learning methods to learn more comprehensive representations of malware variants, which significantly reduces the cost of the entire graph retraining. As a result, MG-DVD is equipped with the ability to detect malware variants in real time, and it presents better interpretability by introducing meaningful meta-graphs. Comprehensive experiments on large-scale samples prove that our proposed MG-DVD outperforms state-of-the-art methods in detecting malware variants in terms of effectiveness and efficiency.

IJCAI Conference 2021 Conference Paper

Recent Advances in Adversarial Training for Adversarial Robustness

• Tao Bai
• Jinqi Luo
• Jun Zhao
• Bihan Wen
• Qian Wang

Adversarial training is one of the most effective approaches for deep learning models to defend against adversarial examples. Unlike other defense strategies, adversarial training aims to enhance the robustness of models intrinsically. During the past few years, adversarial training has been studied and discussed from various aspects, which deserves a comprehensive review. For the first time in this survey, we systematically review the recent progress on adversarial training for adversarial robustness with a novel taxonomy. Then we discuss the generalization problems in adversarial training from three perspectives and highlight the challenges which are not fully tackled. Finally, we present potential future directions.

AAAI Conference 2021 Conference Paper

What the Role is vs. What Plays the Role: Semi-Supervised Event Argument Extraction via Dual Question Answering

• Yang Zhou
• Yubo Chen
• Jun Zhao
• Yin Wu
• Jiexin Xu
• Jinlong Li

Event argument extraction is an essential task in event extraction, and become particularly challenging in the case of low-resource scenarios. We solve the issues in existing studies under low-resource situations from two sides. From the perspective of the model, the existing methods always suffer from the concern of insufficient parameter sharing and do not consider the semantics of roles, which is not conducive to dealing with sparse data. And from the perspective of the data, most existing methods focus on data generation and data augmentation. However, these methods rely heavily on external resources, which is more laborious to create than obtain unlabeled data. In this paper, we propose DualQA, a novel framework, which models the event argument extraction task as question answering to alleviate the problem of data sparseness and leverage the duality of event argument recognition which is to ask “What plays the role”, as well as event role recognition which is to ask “What the role is”, to mutually improve each other. Experimental results on two datasets prove the effectiveness of our approach, especially in extremely low-resource situations.

AAAI Conference 2020 Conference Paper

Capturing Sentence Relations for Answer Sentence Selection with Multi-Perspective Graph Encoding

• Zhixing Tian
• Yuanzhe Zhang
• Xinwei Feng
• Wenbin Jiang
• Yajuan Lyu
• Kang Liu
• Jun Zhao

This paper focuses on the answer sentence selection task. Unlike previous work, which only models the relation between the question and each candidate sentence, we propose Multi-Perspective Graph Encoder (MPGE) to take the relations among the candidate sentences into account and capture the relations from multiple perspectives. By utilizing MPGE as a module, we construct two answer sentence selection models which are based on traditional representation and pre-trained representation, respectively. We conduct extensive experiments on two datasets, WikiQA and SQuAD. The results show that the proposed MPGE is effective for both types of representation. Moreover, the overall performance of our proposed model surpasses the state-of-the-art on both datasets. Additionally, we further validate the robustness of our method by the adversarial examples of AddSent and AddOneSent.

IJCAI Conference 2020 Conference Paper

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

• Zhuang Liu
• Degen Huang
• Kaiyu Huang
• Zhuang Li
• Jun Zhao

There is growing interest in the tasks of financial text mining. Over the past few years, the progress of Natural Language Processing (NLP) based on deep learning advanced rapidly. Significant progress has been made with deep learning showing promising results on financial text mining models. However, as NLP models require large amounts of labeled training data, applying deep learning to financial text mining is often unsuccessful due to the lack of labeled training data in financial fields. To address this issue, we present FinBERT (BERT for Financial Text Mining) that is a domain specific language model pre-trained on large-scale financial corpora. In FinBERT, different from BERT, we construct six pre-training tasks covering more knowledge, simultaneously trained on general corpora and financial domain corpora, which can enable FinBERT model better to capture language knowledge and semantic information. The results show that our FinBERT outperforms all current state-of-the-art models. Extensive experimental results demonstrate the effectiveness and robustness of FinBERT. The source code and pre-trained models of FinBERT are available online.

IJCAI Conference 2020 Conference Paper

Knowledge Enhanced Event Causality Identification with Mention Masking Generalizations

• Jian Liu
• Yubo Chen
• Jun Zhao

Identifying causal relations of events is a crucial language understanding task. Despite many efforts for this task, existing methods lack the ability to adopt background knowledge, and they typically generalize poorly to new, previously unseen data. In this paper, we present a new method for event causality identification, aiming to address limitations of previous methods. On the one hand, our model can leverage external knowledge for reasoning, which can greatly enrich the representation of events; On the other hand, our model can mine event-agnostic, context-specific patterns, via a mechanism called event mention masking generalization, which can greatly enhance the ability of our model to handle new, previously unseen cases. In experiments, we evaluate our model on three benchmark datasets and show our model outperforms previous methods by a significant margin. Moreover, we perform 1) cross-topic adaptation, 2) exploiting unseen predicates, and 3) cross-task adaptation to evaluate the generalization ability of our model. Experimental results show that our model demonstrates a definite advantage over previous methods.

JBHI Journal 2019 Journal Article

Improved False Positive Reduction by Novel Morphological Features for Computer-Aided Polyp Detection in CT Colonography

• Yacheng Ren
• Jingchen Ma
• Junfeng Xiong
• Yi Chen
• Lin Lu
• Jun Zhao

Computer-aided detection (CAD) systems can assist radiologists in reducing the interpretation time and improving the detection results in computed tomographic colonography (CTC). However, existing false positives (FPs) impair the advantages of CAD systems. This study aims to develop new morphological features for the FP reduction while maintaining high detection sensitivity. Volumetric feature maps are computed for each polyp candidate by using three-dimensional (3-D) geodesic distance transformation, circular transformation (CcT), and quantized convergence index (QCI) filters. Then, new morphological features are developed based on the curvature, fractal dimension, and volumetric feature maps. To the best of our knowledge, we are also the first to develop 3-D CcT and QCI filters specifically for colonic polyps. The new morphological features were evaluated to reduce the FPs by using 456 oral contrast-enhanced CT scans from 228 patients with 130 polyps ≥5 mm. For comparison, the well-defined features from our previous work were used to generate a baseline reference. The additional use of the new morphological features reduced the FP rate from 4. 2 to 2. 0 FPs per scan (i. e. , 52. 4% FP reduction percentage) at 96. 2% by-polyp sensitivity and from 4. 5 to 2. 1 FPs per scan (i. e. , 53. 3% FP reduction percentage) at 93. 9% per-scan sensitivity for polyps ≥5 mm. Experimental results indicate that the new morphological features can effectively reduce the FP rate without sacrificing detection sensitivity. We believe that the newly developed morphological features would advance the CAD systems to assist radiologists in interpreting CTC images.

IJCAI Conference 2018 Conference Paper

Curriculum Learning for Natural Answer Generation

• Cao Liu
• Shizhu He
• Kang Liu
• Jun Zhao

By reason of being able to obtain natural language responses, natural answers are more favored in real-world Question Answering (QA) systems. Generative models learn to automatically generate natural answers from large-scale question answer pairs (QA-pairs). However, they are suffering from the uncontrollable and uneven quality of QA-pairs crawled from the Internet. To address this problem, we propose a curriculum learning based framework for natural answer generation (CL-NAG), which is able to take full advantage of the valuable learning data from a noisy and uneven-quality corpus. Specifically, we employ two practical measures to automatically measure the quality (complexity) of QA-pairs. Based on the measurements, CL-NAG firstly utilizes simple and low-quality QA-pairs to learn a basic model, and then gradually learns to produce better answers with richer contents and more complete syntaxes based on more complex and higher-quality QA-pairs. In this way, all valuable information in the noisy and uneven-quality corpus could be fully exploited. Experiments demonstrate that CL-NAG outperforms the state-of-the-arts, which increases 6. 8% and 8. 7% in the accuracy for simple and complex questions, respectively.

AAAI Conference 2018 Conference Paper

Event Detection via Gated Multilingual Attention Mechanism

• Jian Liu
• Yubo Chen
• Kang Liu
• Jun Zhao

Identifying event instance in text plays a critical role in building NLP applications such as Information Extraction (IE) system. However, most existing methods for this task focus only on monolingual clues of a specific language and ignore the massive information provided by other languages. Data scarcity and monolingual ambiguity hinder the performance of these monolingual approaches. In this paper, we propose a novel multilingual approach — dubbed as Gated MultiLingual Attention (GMLATT) framework — to address the two issues simultaneously. In specific, to alleviate data scarcity problem, we exploit the consistent information in multilingual data via context attention mechanism. Which takes advantage of the consistent evidence in multilingual data other than learning only from monolingual data. To deal with monolingual ambiguity problem, we propose gated cross-lingual attention to exploit the complement information conveyed by multilingual data, which is helpful for the disambiguation. The cross-lingual attention gate serves as a sentinel modelling the confidence of the clues provided by other languages and controls the information integration of various languages. We have conducted extensive experiments on the ACE 2005 benchmark. Experimental results show that our approach significantly outperforms state-of-the-art methods.

AAAI Conference 2018 Conference Paper

Large Scaled Relation Extraction With Reinforcement Learning

• Xiangrong Zeng
• Shizhu He
• Kang Liu
• Jun Zhao

Sentence relation extraction aims to extract relational facts from sentences, which is an important task in natural language processing field. Previous models rely on the manually labeled supervised dataset. However, the human annotation is costly and limits to the number of relation and data size, which is difficult to scale to large domains. In order to conduct largely scaled relation extraction, we utilize an existing knowledge base to heuristically align with texts, which not rely on human annotation and easy to scale. However, using distant supervised data for relation extraction is facing a new challenge: sentences in the distant supervised dataset are not directly labeled and not all sentences that mentioned an entity pair can represent the relation between them. To solve this problem, we propose a novel model with reinforcement learning. The relation of the entity pair is used as distant supervision and guide the training of relation extractor with the help of reinforcement learning method. We conduct two types of experiments on a publicly released dataset. Experiment results demonstrate the effectiveness of the proposed method compared with baseline models, which achieves 13. 36% improvement.

IJCAI Conference 2017 Conference Paper

Conditional Generative Adversarial Networks for Commonsense Machine Comprehension

• Bingning Wang
• Kang Liu
• Jun Zhao

Recently proposed Story Cloze Test [Mostafazadeh et al. , 2016] is a commonsense machine comprehension application to deal with natural language understanding problem. This dataset contains a lot of story tests which require commonsense inference ability. Unfortunately, the training data is almost unsupervised where each context document followed with only one positive sentence that can be inferred from the context. However, in the testing period, we must make inference from two candidate sentences. To tackle this problem, we employ the generative adversarial networks (GANs) to generate fake sentence. We proposed a Conditional GANs in which the generator is conditioned by the context. Our experiments show the advantage of the CGANs in discriminating sentence and achieve state-of-the-art results in commonsense story reading comprehension task compared with previous feature engineering and deep learning methods.

AAAI Conference 2017 Conference Paper

Distant Supervision for Relation Extraction with Sentence-Level Attention and Entity Descriptions

• Guoliang Ji
• Kang Liu
• Shizhu He
• Jun Zhao

Distant supervision for relation extraction is an efficient method to scale relation extraction to very large corpora which contains thousands of relations. However, the existing approaches have flaws on selecting valid instances and lack of background knowledge about the entities. In this paper, we propose a sentence-level attention model to select the valid instances, which makes full use of the supervision information from knowledge bases. And we extract entity descriptions from Freebase and Wikipedia pages to supplement background knowledge for our task. The background knowledge not only provides more information for predicting relations, but also brings better entity representations for the attention module. We conduct three experiments on a widely used dataset and the experimental results show that our approach outperforms all the baseline systems significantly.

AAAI Conference 2016 Conference Paper

A Joint Model for Question Answering over Multiple Knowledge Bases

• Yuanzhe Zhang
• Shizhu He
• Kang Liu
• Jun Zhao

As the amount of knowledge bases (KBs) grows rapidly, the problem of question answering (QA) over multiple KBs has drawn more attention. The most significant distinction between multiple KB-QA and single KB-QA is that the former must consider the alignments between KBs. The pipeline strategy first constructs the alignments independently, and then uses the obtained alignments to construct queries. However, alignment construction is not a trivial task, and the introduced noises would be passed on to query construction. By contrast, we notice that alignment construction and query construction are interactive steps, and jointly considering them would be beneficial. To this end, we present a novel joint model based on integer linear programming (ILP), uniting these two procedures into a uniform framework. The experimental results demonstrate that the proposed approach outperforms state-of-the-art systems, and is able to improve the performance of both alignment construction and query construction.

AAAI Conference 2016 Conference Paper

A Probabilistic Soft Logic Based Approach to Exploiting Latent and Global Information in Event Classification

• Shulin Liu
• Kang Liu
• Shizhu He
• Jun Zhao

Global information such as event-event association, and latent local information such as fine-grained entity types1, are crucial to event classification. However, existing methods typically focus on sophisticated local features such as part-ofspeech tags, either fully or partially ignoring the aforementioned information. By contrast, this paper focuses on fully employing them for event classification. We notice that it is difficult to encode some global information such as eventevent association for previous methods. To resolve this problem, we propose a feasible approach which encodes global information in the form of logic using Probabilistic Soft Logic model. Experimental results show that, our proposed approach advances state-of-the-art methods, and achieves the best F1 score to date on the ACE data set.

IJCAI Conference 2016 Conference Paper

Employing External Rich Knowledge for Machine Comprehension

• Bingning Wang
• Shangmin Guo
• Kang Liu
• Shizhu He
• Jun Zhao

Recently proposed machine comprehension (MC) application is an effort to deal with natural language understanding problem. However, the small size of machine comprehension labeled data confines the application of deep neural networks architectures that have shown advantage in semantic inference tasks. Previous methods use a lot of NLP tools to extract linguistic features but only gain little improvement over simple baseline. In this paper, we build an attention-based recurrent neural network model, train it with the help of external knowledge which is semantically relevant to machine comprehension, and achieves a new state-of-art result.

IS Journal 2016 Journal Article

How to Generate a Good Word Embedding

• Siwei Lai
• Kang Liu
• Shizhu He
• Jun Zhao

The authors analyze three critical components in training word embeddings: model, corpus, and training parameters. They systematize existing neural-network-based word embedding methods and experimentally compare them using the same corpus. They then evaluate each word embedding in three ways: analyzing its semantic properties, using it as a feature for supervised tasks, and using it to initialize neural networks. They also provide several simple guidelines for training good word embeddings.

AAAI Conference 2016 Conference Paper

Knowledge Graph Completion with Adaptive Sparse Transfer Matrix

• Guoliang Ji
• Kang Liu
• Shizhu He
• Jun Zhao

We model knowledge graphs for their completion by encoding each entity and relation into a numerical space. All previous work including Trans(E, H, R, and D) ignore the heterogeneity (some relations link many entity pairs and others do not) and the imbalance (the number of head entities and that of tail entities in a relation could be different) of knowledge graphs. In this paper, we propose a novel approach TranSparse to deal with the two issues. In TranSparse, transfer matrices are replaced by adaptive sparse matrices, whose sparse degrees are determined by the number of entities (or entity pairs) linked by relations. In experiments, we design structured and unstructured sparse patterns for transfer matrices and analyze their advantages and disadvantages. We evaluate our approach on triplet classification and link prediction tasks. Experimental results show that TranSparse outperforms Trans(E, H, R, and D) significantly, and achieves state-of-the-art performance.

IJCAI Conference 2015 Conference Paper

A Subspace Learning Framework for Cross-Lingual Sentiment Classification with Partial Parallel Data

• Guangyou Zhou
• Tingting He
• Jun Zhao
• Wensheng Wu

Cross-lingual sentiment classification aims to automatically predict sentiment polarity (e. g. , positive or negative) of data in a label-scarce target language by exploiting labeled data from a label-rich language. The fundamental challenge of cross-lingual learning stems from a lack of overlap between the feature spaces of the source language data and that of the target language data. To address this challenge, previous work in the literature mainly relies on the large amount of bilingual parallel corpora to bridge the language gap. In many real applications, however, it is often the case that we have some partial parallel data but it is an expensive and time-consuming job to acquire large amount of parallel data on different languages. In this paper, we propose a novel subspace learning framework by leveraging the partial parallel data for cross-lingual sentiment classification. The proposed approach is achieved by jointly learning the document-aligned review data and un-aligned data from the source language and the target language via a non-negative matrix factorization framework. We conduct a set of experiments with cross-lingual sentiment classification tasks on multilingual Amazon product reviews. Our experimental results demonstrate the efficacy of the proposed cross-lingual approach.

IJCAI Conference 2015 Conference Paper

Convolutional Neural Networks for Text Hashing

• Jiaming Xu
• Peng Wang
• Guanhua Tian
• Bo Xu
• Jun Zhao
• Fangyuan Wang
• Hongwei Hao

Hashing, as a popular approximate nearest neighbor search, has been widely used for large-scale similarity search. Recently, a spectrum of machine learning methods are utilized to learn similarity-preserving binary codes. However, most of them directly encode the explicit features, keywords, which fail to preserve the accurate semantic similarities in binary code beyond keyword matching, especially on short texts. Here we propose a novel text hashing framework with convolutional neural networks. In particular, we first embed the keyword features into compact binary code with a locality preserving constraint. Meanwhile word features and position features are together fed into a convolutional network to learn the implicit features which are further incorporated with the explicit features to fit the pre-trained binary code. Such base method can be successfully accomplished without any external tags/labels, and other three model variations are designed to integrate tags/labels. Experimental results show the superiority of our proposed approach over several state-of-the-art hashing methods when tested on one short text dataset as well as one normal text dataset.

IS Journal 2015 Journal Article

Question Answering over Knowledge Bases

• Kang Liu
• Jun Zhao
• Shizhu He
• Yuanzhe Zhang

Question answering over knowledge bases is a challenging task for next-generation search engines. The core of this task is to understand the meaning of questions and translate them into structured language-based queries. Previous research has focused on a specific knowledge base with a constrained domain, but with the increase in the size and domain of existing knowledge bases, fulfilling this aim is even more challenging. This article introduces the mainstream methods for question answering over knowledge bases, describing typical semantic meaning representation models and state-of-the-art systems for converting questions to predefined logical forms. It also puts a particular focus on the approaches for question answering over a large-scale knowledge base and multiple heterogeneous knowledge bases.

AAAI Conference 2015 Conference Paper

Recurrent Convolutional Neural Networks for Text Classification

• Siwei Lai
• Liheng Xu
• Kang Liu
• Jun Zhao

Text classification is a foundational task in many NLP applications. Traditional text classifiers often rely on many human-designed features, such as dictionaries, knowledge bases and special tree kernels. In contrast to traditional methods, we introduce a recurrent convolutional neural network for text classification without human-designed features. In our model, we apply a recurrent structure to capture contextual information as far as possible when learning word representations, which may introduce considerably less noise compared to traditional window-based neural networks. We also employ a max-pooling layer that automatically judges which words play key roles in text classification to capture the key components in texts. We conduct experiments on four commonly used datasets. The experimental results show that the proposed method outperforms the state-of-the-art methods on several datasets, particularly on document-level datasets.

IJCAI Conference 2013 Conference Paper

Cross Lingual Entity Linking with Bilingual Topic Model

• Tao Zhang
• Kang Liu
• Jun Zhao

Cross lingual entity linking means linking an entity mention in a background source document in one language with the corresponding real world entity in a knowledge base written in the other language. The key problem is to measure the similarity score between the context of the entity mention and the document of the candidate entity. This paper presents a general framework for doing cross lingual entity linking by leveraging a large scale and bilingual knowledge base, Wikipedia. We introduce a bilingual topic model that mining bilingual topic from this knowledge base with the assumption that the same Wikipedia concept documents of two different languages share the same semantic topic distribution. The extracted topics have two types of representation, with each type corresponding to one language. Thus both the context of the entity mention and the document of the candidate entity can be represented in a space using the same semantic topics. We use these topics to do cross lingual entity linking. Experimental results show that the proposed approach can obtain the competitive results compared with the state-of-art approach.

IJCAI Conference 2013 Conference Paper

Improving Question Retrieval in Community Question Answering Using World Knowledge

• Guangyou Zhou
• Yang Liu
• Fang Liu
• Daojian Zeng
• Jun Zhao

Community question answering (cQA), which provides a platform for people with diverse background to share information and knowledge, has become an increasingly popular research topic. In this paper, we focus on the task of question retrieval. The key problem of question retrieval is to measure the similarity between the queried questions and the historical questions which have been solved by other users. The traditional methods measure the similarity based on the bag-of-words (BOWs) representation. This representation neither captures dependencies between related words, nor handles synonyms or polysemous words. In this work, we first propose a way to build a concept thesaurus based on the semantic relations extracted from the world knowledge of Wikipedia. Then, we develop a unified framework to leverage these semantic relations in order to enhance the question similarity in the concept space. Experiments conducted on a real cQA data set show that with the help of Wikipedia thesaurus, the performance of question retrieval is improved as compared to the traditional methods.

IJCAI Conference 2013 Conference Paper

Opinion Target Extraction Using Partially-Supervised Word Alignment Model

• Kang Liu
• Liheng Xu
• Yang Liu
• Jun Zhao

Mining opinion targets from online reviews is an important and challenging task in opinion mining. This paper proposes a novel approach to extract opinion targets by using partially-supervised word alignment model (PSWAM). At first, we apply PSWAM in a monolingual scenario to mine opinion relations in sentences and estimate the associations between words. Then, a graph-based algorithm is exploited to estimate the confidence of each candidate, and the candidates with higher confidence will be extracted as the opinion targets. Compared with existing syntax-based methods, PSWAM can effectively avoid parsing errors when dealing with informal sentences in online reviews. Compared with the methods using alignment model, PSWAM can capture opinion relations more precisely through partial supervision from partial alignment links. Moreover, when estimating candidate confidence, we make penalties on higherdegree vertices in our graph-based algorithm in order to decrease the probability of the random walk running into the unrelated regions in the graph. As a result, some errors can be avoided. The experimental results on three data sets with different sizes and languages show that our approach outperforms state-of-the-art methods.

IROS Conference 2009 Conference Paper

Detection of non-flat ground surfaces using V-Disparity images

• Jun Zhao
• Mark Albert Whitty
• Jay Katupitiya

Ground plane detection plays an important role in stereo vision based obstacle detection methods. Recently, V-disparity image has been widely used for ground plane detection. The existing approach based on V-disparity image can detect flat ground successfully but have difficulty in detecting non-flat ground. In this paper, we discuss the representation of non-flat ground in V-disparity image, based on which we propose a method to detect non-flat ground using V-disparity image.

IJCAI Conference 2007 Conference Paper

• Xiangyu Duan
• Jun Zhao
• Bo Xu

Currently most word sense disambiguation (WSD) systems are relatively individual word sense experts. Scarcely do these systems take word sense transitions between senses of linearly consecutive words or syntactically dependent words into consideration. Word sense transitions are very important. They embody the fluency of semantic expression and avoid sparse data problem effectively. In this paper, HowNet knowledge base is used to decompose every word sense into several sememes. Then one transition between two words' senses becomes multiple transitions between sememes. Sememe transitions are much easier to be captured than word sense transitions due to much less sememes. When sememes are labeled, WSD is done. In this paper, multi-layered conditional random fields (MLCRF) is proposed to model sememe transitions. The experiments show that MLCRF performs better than a base-line system and a maximum entropy model. Syntactic and hypernym features can enhance the performance significantly.

ICRA Conference 2007 Conference Paper

Global Correlation Based Ground Plane Estimation Using V-Disparity Image

• Jun Zhao
• Jay Katupitiya
• James Ward

This paper presents the estimation of the position of the ground plane for navigation of on-road or off-road vehicles, in particular for obstacle detection using stereo vision. Ground plane estimation plays an important role in stereo vision based obstacle detection tasks. V-disparity image is widely used for ground plane estimation. However, it heavily relies on distinct road features which may not exist. In here, we introduce a global correlation method to extract the position of the ground plane in V-disparity image even without distinct road features.

IROS Conference 2006 Conference Paper

A Dynamic Programming Approach Based Stereo Vision Algorithm Improving Object Border Performance

• Jun Zhao
• Jay Katupitiya

The key issue of stereo vision algorithm using dynamic programming approach is to establish the cost function. Normally the cost of a match is the matching difference or matching error. This is almost the same for different algorithms in this approach. The only difference is in the cost of occlusion. Actually the cost of a known occluded point should be zero, because it is a good match. The difficulty is how to find the occluded region. In this paper, we present a new algorithm in which we first find the occluded region in the image using correlation approach. Then we establish a cost function in which the cost of the occluded region is set to zero. Then we use dynamic programming approach to minimize the energy of this cost function. Experimental results show that this algorithm can achieve excellent results with acceptable time efficiency

IROS Conference 2006 Conference Paper

A Fast Stereo Vision Algorithm With Improved Performance at Object Borders

• Jun Zhao
• Jay Katupitiya

Conventional correlation based stereo vision algorithms have poor performance at object borders due to occlusion. This paper analyzes image characteristics at depth discontinuities to improve the performance. It uses a new window scheme, especially to handle occlusion. The results show dramatically improved performance at depth discontinuities

IROS Conference 2005 Conference Paper

On-tree fruit recognition using texture properties and color data

• Jun Zhao
• Joel Tow
• Jay Katupitiya

As a prelude to using stereo vision to accurately locate apples in an orchard, this paper presents a vision based algorithm to locate apples in a single image. On-tree situations of contrasting red and green apples as well as green apples in the orchard with poor contrast have been considered. The study found out that the redness in both cases of red and green apples can be used to differentiate apples from the rest of the orchard. Texture based edge detection has been combined with redness measures, and area thresholding followed by circle fitting, to determine the location of apples in the image plane. In the case of severely cluttered environments, Laplacian filters have been used to further clutter the foliage arrays by edge enhancement so that texture differences between the foliage and the apples increased thereby facilitating the separation of apples from the foliage. Results are presented that show the recognition of red and green apples in a number of situations as well as apples that are clustered together and/or occluded.