Arrow Research search

Author name cluster

Jian Zhao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

28 papers
2 author rows

Possible papers

28

AAAI Conference 2026 Conference Paper

Bias-Restrained Prefix Representation Finetuning for Mathematical Reasoning

  • Sirui Liang
  • Pengfei Cao
  • Jian Zhao
  • Cong Huang
  • Jun Zhao
  • Kang Liu

Parameter-Efficient finetuning (PEFT) enhances model performance on downstream tasks by updating a minimal subset of parameters. Representation finetuning (ReFT) methods further improve efficiency by freezing model weights and optimizing internal representations with fewer parameters than PEFT, outperforming PEFT on several tasks. However, ReFT exhibits a significant performance decline on mathematical reasoning tasks. To address this problem, the paper demonstrates that ReFT's poor performance on mathematical tasks primarily stems from its struggle to generate effective reasoning prefixes during the early inference phase. Moreover, ReFT disturbs the numerical encoding and the error accumulats during the CoT stage. Based on these observations, this paper proposes Bias-REstrained Prefix Representation FineTuning (BREP ReFT), which enhances ReFT's mathematical reasoning capability by truncating training data to optimize the generation of initial reasoning prefixes, intervening on the early inference stage to prevent error accumulation, and constraining the intervention vectors' magnitude to avoid disturbing numerical encoding. Extensive experiments across diverse model architectures demonstrate BREP's superior effectiveness, efficiency, and robust generalization capability, outperforming both standard ReFT and weight-based PEFT methods on the task of mathematical reasoning.

PDF Details DOI

AAAI Conference 2026 Conference Paper

From Chaos to Cure: A Prefix Heuristics Guided Model-Agnostic Adaptive Detoxification Framework

  • Yuhu Shang
  • Xiang Cheng
  • Yimeng Ren
  • Huijia Wu
  • Xuexiong Luo
  • Kangkang Lu
  • Jian Zhao
  • Zhaofeng He

The impressive performance of large language models (LLMs) also brings inherent toxicity risks, prompting the need for effective detoxification to support responsible deployment. Prevailing methods generally follow an inflexible model-specific fashion, addressing only individual models or model families. Moreover, overlooking the underlying toxic risks involved in the input prefix can lead to toxic accumulation during autoregressive generation. Existing methods rely on external strong attribute interventions to address this issue, which further exacerbates contextual semantic inconsistencies and makes it difficult to balance toxicity efficacy and generation quality. To address these concerns, we propose a novel Model-Agnostic Adaptive Detoxification (MAAD) framework. To address accumulating toxicity, we present prefix heuristics that serve as contextual signals, guiding the base LLM toward safer generation. Along this line, we construct an antidote dataset to support a lightweight model, Detoxifier, which steers the base LLM to make in-scope and reliable detoxifying distribution adjustments while preserving fluency and contextual understanding. Designed as an easy-to-deploy module, Detoxifier requires a small amount of data and can be seamlessly applied to various base LLMs with one-off training. Since over-purifying often reduces diversity, we also propose a dynamic truncation method called CW-cutoff sampling to trade off language model quality and diversity. Extensive experiments demonstrate that MAAD strikes a better balance between detoxification effectiveness and generation quality, while also maintaining model utility.

PDF Details DOI

AAAI Conference 2026 Conference Paper

GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

  • Jian Zhao
  • Runze Liu
  • Kaiyan Zhang
  • Zhimu Zhou
  • Junqi Gao
  • Dong Li
  • Jiafei Lyu
  • Zhouyi Qian

Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs.

PDF Details DOI

AAMAS Conference 2026 Conference Paper

LLM-SMAC: Solving Multi-Agent Decision-Making Tasks via LLM Decision Tree Code Generation

  • Yue Deng
  • Weiyu Ma
  • Yuxin Fan
  • Ruyi Song
  • Yin Zhang
  • Haifeng Zhang
  • Jian Zhao

StarCraft Multi-Agent Challenge (SMAC) has become a widely used benchmark in multi-agent systems, where agents must control allied units to defeat enemy forces. Traditional MARL methods typically require millions of environment interactions to train parametric policies, which are often non-interpretable and exhibit limited transferability. In this paper, we introduce LLM-SMAC, a closedloop Planner–Coder–Critic framework. Given task descriptions, the LLM planner first generates a decision-tree strategy, which is translated into executable code by the coder. The generated scripts are executed in the environment, and reward signals and runtime feedback are fed back to a critic module for self-reflection and iterative refinement. Through this closed-loop process, this mechanism progressively improves both strategy design and code implementation without large-scale environment exploration. We evaluate our method on the original SMAC tasks and the results show that LLM- SMAC can produce high-quality, interpretable decision trees with minimal interaction, while demonstrating strong transferability across homogeneous SMAC environments without modification.

PDF

AAMAS Conference 2026 Conference Paper

Sim2Sea: Sim-to-Real Policy Transfer for Maritime Vessel Navigation in Congested Waters

  • Xinyu Cui
  • Xuanfa Jin
  • Xue Yan
  • Yongcheng Zeng
  • Luoyang Sun
  • Siying Wei
  • Ruizhi Zhang
  • Jian Zhao

Autonomous navigation in congested maritime environments is a critical capability for a wide range of real-world applications. However, it remains an unresolved challenge due to complex vessel interactions and significant environmental uncertainties. Existing methods often fail in practical deployment due to a substantial simto-real gap, which stems from imprecise simulation, inadequate situational awareness, and unsafe exploration strategies. To address these, we propose Sim2Sea, a comprehensive framework designed to bridge simulation and real-world execution. Sim2Sea advances in three key aspects. First, we develop a GPU-accelerated parallel simulator for scalable and accurate maritime scenario simulation. Second, wedesignadual-streamspatiotemporalpolicythathandles complex dynamics and multi-modal perception, augmented with a velocity-obstacle-guided action masking mechanism to ensure safe and efficient exploration. Finally, a targeted domain randomization scheme helps bridge the sim-to-real gap. Simulation results demonstrate that our method achieves faster convergence and safer trajectories than established baselines. In addition, our policy trained purely in simulation successfully transfers zero-shot to a 17-ton unmanned vessel operating in real-world congested waters. These results validate the effectiveness of Sim2Sea in achieving reliable sim-to-real transfer for practical autonomous maritime navigation. ∗Corresponding to Haifeng Zhang. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). https: //doi. org/10. 65109/CMRP6518

PDF

AAAI Conference 2026 Conference Paper

When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models

  • Wei Cai
  • Shujuan Liu
  • Jian Zhao
  • Ziyan Shi
  • Yusheng Zhao
  • Yuchen Yuan
  • Tianle Zhang
  • Chi Zhang

Multimodal Large Language Models (MLLMs) are susceptible to the implicit reasoning risk, wherein innocuous unimodal inputs synergistically assemble into risky multimodal data that produce harmful outputs. We attribute this vulnerability to the difficulty of MLLMs maintaining safety alignment through long-chain reasoning.To address this issue, we introduce Safe-Semantics-but-Unsafe-Interpretation (SSUI), the first dataset featuring interpretable reasoning paths tailored for such a cross-modal challenge.A novel training framework, Safety-aware Reasoning Path Optimization (SRPO), is also designed based on the SSUI dataset to align the MLLM's internal reasoning process with human safety values. Experimental results show that our SRPO-trained models achieve state-of-the-art results on key safety benchmarks, including the proposed Reasoning Path Benchmark (RSBench), significantly outperforming both open-source and top-tier commercial MLLMs.

PDF Details DOI

IROS Conference 2025 Conference Paper

Bimanual Robot-Assisted Dressing: A Spherical Coordinate-Based Strategy for Tight-Fitting Garments

  • Jian Zhao
  • Yunlong Lian
  • Andy M. Tyrrell
  • Michael Gienger
  • Jihong Zhu

Robot-assisted dressing is a popular but challenging topic in the field of robotic manipulation, offering significant potential to improve the quality of life for individuals with mobility limitations. Currently, the majority of research on robot-assisted dressing focuses on how to put on loose-fitting clothing, with little attention paid to tight garments. For the former, since the armscye is larger, a single robotic arm can usually complete the dressing task successfully. However, for the latter, dressing with a single robotic arm often fails due to the narrower armscye and the property of diminishing rigidity in the armscye, which eventually causes the armscye to get stuck. This paper proposes a bimanual dressing strategy suitable for dressing tight-fitting clothing. To facilitate the encoding of dressing trajectories that adapt to different human arm postures, a spherical coordinate system for dressing is established. We uses the azimuthal angle of the spherical coordinate system as a task-relevant feature for bimanual manipulation. Based on this new coordinate, we employ Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR) for imitation learning of bimanual dressing trajectories, generating dressing strategies that adapt to different human arm postures. The effectiveness of the proposed method is validated through various experiments.

Details

AAAI Conference 2025 Conference Paper

CraftFactory: A Conditioned Control Policy Benchmark for Compositional Generalization

  • Jinbing Hou
  • Youpeng Zhao
  • Jian Zhao

Humans excel at understanding and reasoning about novel, compositionally structured knowledge, largely due to their capacity for compositional generalization—a cognitive skill that has recently been validated in structured neural networks. However, most existing research has focused primarily on semantic translation within canonical language environments, often neglecting the explicit connection to compositional generalization behavior. In contrast, humans typically demonstrate this ability through interaction with their environments rather than solely through internal reasoning. To address this gap, we propose CraftFactory, a benchmark designed for evaluating compositional generalization in an interactive control environment. This benchmark introduces a new challenge for testing compositional generalization in a more realistic and comprehensive manner. CraftFactory stands out due to three key features: (1) it offers an open-ended interactive control environment with thousands of items and flexible actions; (2) it requires advanced compositional inference through various combinations and complex permutations of instructions; and (3) it evaluates compositional generalization intuitively through interactive behavior. By leveraging CraftFactory, we aim to promote the development of more advanced compositional generalization methods, thereby contributing to the broader field of general AI.

PDF Details DOI

ICLR Conference 2025 Conference Paper

INS: Interaction-aware Synthesis to Enhance Offline Multi-agent Reinforcement Learning

  • Yuqian Fu
  • Yuanheng Zhu
  • Jian Zhao
  • Jiajun Chai
  • Dongbin Zhao

Data scarcity in offline multi-agent reinforcement learning (MARL) is a key challenge for real-world applications. Recent advances in offline single-agent reinforcement learning (RL) demonstrate the potential of data synthesis to mitigate this issue. However, in multi-agent systems, interactions between agents introduce additional challenges. These interactions complicate the synthesis of multi-agent datasets, leading to data distortion when inter-agent interactions are neglected. Furthermore, the quality of the synthetic dataset is often constrained by the original dataset. To address these challenges, we propose **INteraction-aware Synthesis (INS)**, which synthesizes high-quality multi-agent datasets using diffusion models. Recognizing the sparsity of inter-agent interactions, INS employs a sparse attention mechanism to capture these interactions, ensuring that the synthetic dataset reflects the underlying agent dynamics. To overcome the limitation of diffusion models requiring continuous variables, INS implements a bit action module, enabling compatibility with both discrete and continuous action spaces. Additionally, we incorporate a select mechanism to prioritize transitions with higher estimated values, further enhancing the dataset quality. Experimental results across multiple datasets in MPE and SMAC environments demonstrate that INS consistently outperforms existing methods, resulting in improved downstream policy performance and superior dataset metrics. Notably, INS can synthesize high-quality data using only 10% of the original dataset, highlighting its efficiency in data-limited scenarios.

Details

EAAI Journal 2024 Journal Article

A high-efficiency local and global detector for diatom-based drowning diagnosis

  • Jiehang Deng
  • Jianfa Yang
  • Haomin Wei
  • Guosheng Gu
  • Qingqing Xiang
  • Yukun Du
  • Chao Liu
  • Zhiqiang Chen

Detection and classification of diatoms has been considered as the most effective method for the drowning diagnosis in forensic practice. However, challenges (such as limited samples, complex background interference, and detecting multiple objects) remain during detection and classification. To address these challenges, we propose a MDMS (multi-scale dynamic multi-head self-attention)-based deep learning network for the diatom-based drowning diagnosis. This framework uses TFFT (two-stage full fine-tuning training strategy) to solve the challenges of limited diatom samples and detecting multiple objects, and builds a multi-scale dynamic multi-head self-attention module to extract the local and global features of diatom images. In the meantime, we introduce an online hard example mining strategy to attenuate the complex background interference. The experimental results show that the proposed framework can effectively reduce the missing and false detection rates of diatom objects, with the mAP (mean Average Precision) reaching 92. 434%, which is better than the result using the mainstream methods.

Details DOI

EAAI Journal 2024 Journal Article

Cascade fusion of multi-modal and multi-source feature fusion by the attention for three-dimensional object detection

  • Fengning Yu
  • Jing Lian
  • Linhui Li
  • Jian Zhao

Three dimensional (3D) object detection using the camera and light detection and ranging (LiDAR) fusion model has received much attention for meeting the environmental sensing requirement in autonomous vehicles. Due to the intrinsic data difference and the uneven distribution of LiDAR data, integrating the camera and LiDAR to solve the problem of the uneven spatial distribution of LiDAR data is still challenging. This paper proposes a novel 3D detector, which decorates the point cloud voxels with the semantic features at multi-scale and fuses the multi-source features by cross-attention to improve the detection performance. Specifically, the multi-modal fusion aims to maintain the cross-modal consistency by using the frustum model to construct the correspondence between point voxels and image pixels at multi-scale. To fully exploit the geometric constraints of voxels, we developed a adaptive sampling radius module to dynamically select the sampling radius of the voxel set abstraction module based on softmax. Moreover, we propose a multi-source fusion module which utilizes the cross-attention and takes the raw point cloud’s spatial distribution as the clue to fuse the features of point clouds, bird’s eye view information and the aggregated voxels to obtain the multi-scale and multi-source fusion features. Finally, region proposal network is adopted to generate and refine the 3D bbox and class prediction based on the fusion feature and bird’s eye view feature. Extensive experiments are conducted on the two publicly available 3D object datasets proposed by Karlsruhe Institute of technology and Toyota Technological Institute (KITTI) and the nuScenes. The proposed model reaches 86. 27% mean Average Precision (mAP) for car on KITTI and obtains 6. 56% gains than PointPainting. Moreover, our model reaches 70. 70% mAP on nuScens and improves 2. 61% than Transfusion. Furthermore, comprehensive ablation experiments are conducted to validate the effects and contribution of the different components of the proposed model.

Details DOI

NeurIPS Conference 2024 Conference Paper

Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model

  • Yifan Duan
  • Jian Zhao
  • Junyuan Mao
  • Hao Wu
  • Jingyu Xu
  • Shilong Wang
  • Caoyuan Ma
  • Kai Wang

Spatio-temporal (ST) prediction has garnered a De facto attention in earth sciences, such as meteorological prediction, human mobility perception. However, the scarcity of data coupled with the high expenses involved in sensor deployment results in notable data imbalances. Furthermore, models that are excessively customized and devoid of causal connections further undermine the generalizability and interpretability. To this end, we establish a causal framework for ST predictions, termed CaPaint, which targets to identify causal regions in data and endow model with causal reasoning ability in a two-stage process. Going beyond this process, we utilize the back-door adjustment to specifically address the sub-regions identified as non-causal in the upstream phase. Specifically, we employ a novel image inpainting technique. By using a fine-tuned unconditional Diffusion Probabilistic Model (DDPM) as the generative prior, we in-fill the masks defined as environmental parts, offering the possibility of reliable extrapolation for potential data distributions. CaPaint overcomes the high complexity dilemma of optimal ST causal discovery models by reducing the data generation complexity from exponential to quasi-linear levels. Extensive experiments conducted on five real-world ST benchmarks demonstrate that integrating the CaPaint concept allows models to achieve improvements ranging from 4. 3% to 77. 3%. Moreover, compared to traditional mainstream ST augmenters, CaPaint underscores the potential of diffusion models in ST enhancement, offering a novel paradigm for this field. Our project is available at https: //anonymous. 4open. science/r/12345-DFCC.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model

  • Ziqi Xie
  • Weidong Zhao
  • Xianhui Liu
  • Jian Zhao
  • Ning Jia

Deep learning-based image stitching pipelines are typically divided into three cascading stages: registration, fusion, and rectangling. Each stage requires its own network training and is tightly coupled to the others, leading to error propagation and posing significant challenges to parameter tuning and system stability. This paper proposes the Simple and Robust Stitcher (SRStitcher), which revolutionizes the image stitching pipeline by simplifying the fusion and rectangling stages into a unified inpainting model, requiring no model training or fine-tuning. We reformulate the problem definitions of the fusion and rectangling stages and demonstrate that they can be effectively integrated into an inpainting task. Furthermore, we design the weighted masks to guide the reverse process in a pre-trained large-scale diffusion model, implementing this integrated inpainting task in a single inference. Through extensive experimentation, we verify the interpretability and generalization capabilities of this unified model, demonstrating that SRStitcher outperforms state-of-the-art methods in both performance and stability.

PDF Details DOI

AAAI Conference 2024 Conference Paper

RoPDA: Robust Prompt-Based Data Augmentation for Low-Resource Named Entity Recognition

  • Sihan Song
  • Furao Shen
  • Jian Zhao

Data augmentation has been widely used in low-resource NER tasks to tackle the problem of data sparsity. However, previous data augmentation methods have the disadvantages of disrupted syntactic structures, token-label mismatch, and requirement for external knowledge or manual effort. To address these issues, we propose Robust Prompt-based Data Augmentation (RoPDA) for low-resource NER. Based on pre-trained language models (PLMs) with continuous prompt, RoPDA performs entity augmentation and context augmentation through five fundamental augmentation operations to generate label-flipping and label-preserving examples. To optimize the utilization of the augmented samples, we present two techniques: self-consistency filtering and mixup. The former effectively eliminates low-quality samples with a bidirectional mask, while the latter prevents performance degradation arising from the direct utilization of labelflipping samples. Extensive experiments on three popular benchmarks from different domains demonstrate that RoPDA significantly improves upon strong baselines, and also outperforms state-of-the-art semi-supervised learning methods when unlabeled data is included.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Unified Single-Stage Transformer Network for Efficient RGB-T Tracking

  • Jianqiang Xia
  • Dianxi Shi
  • Ke Song
  • Linna Song
  • Xiaolei Wang
  • Songchang Jin
  • Chenran Zhao
  • Yu Cheng

Most existing RGB-T tracking networks extract modality features in a separate manner, which lacks interaction and mutual guidance between modalities. This limits the network's ability to adapt to the diverse dual-modality appearances of targets and the dynamic relationships between the modalities. Additionally, the three-stage fusion tracking paradigm followed by these networks significantly restricts the tracking speed. To overcome these problems, we propose a unified single-stage Transformer RGB-T tracking network, namely USTrack, which unifies the above three stages into a single ViT (Vision Transformer) backbone through joint feature extraction, fusion and relation modeling. With this structure, the network can not only extract the fusion features of templates and search regions under the interaction of modalities, but also significantly improve tracking speed through the single-stage fusion tracking paradigm. Furthermore, we introduce a novel feature selection mechanism based on modality reliability to mitigate the influence of invalid modalities for final prediction. Extensive experiments on three mainstream RGB-T tracking benchmarks show that our method achieves the new state-of-the-art while achieving the fastest tracking speed of 84. 2FPS. Code is available at https: //github. com/xiajianqiang/USTrack.

PDF Details DOI

TIST Journal 2023 Journal Article

3D-Guided Frontal Face Generation for Pose-Invariant Recognition

  • Hao Wu
  • Jianyang Gu
  • Xiaojin Fan
  • He Li
  • Lidong Xie
  • Jian Zhao

Although deep learning techniques have achieved extraordinary accuracy in recognizing human faces, the pose variances of images captured in real-world scenarios still hinder reliable model appliance. To mitigate this gap, we propose to recognize faces via generation frontal face images with a 3D -Guided Deep P ose- I nvariant Face Recognition M odel (3D-PIM) consisted of a simulator and a refiner module. The simulator employs a 3D Morphable Model (3D MM) to fit the shape and appearance features and recover primary frontal images with less training data. The refiner further enhances the image realism on both global facial structure and local details with adversarial training, while keeping the discriminative identity information consistent with original images. An Adaptive Weighting (AW) metric is then adopted to leverage the complimentary information from recovered frontal faces and original profile faces and to obtain credible similarity scores for recognition. Extended experiments verify the superiority of the proposed “recognition via generation” framework over state-of-the-art.

Details DOI

TIST Journal 2023 Journal Article

A Joint Entity and Relation Extraction Model based on Efficient Sampling and Explicit Interaction

  • Qibin Li
  • Nianmin Yao
  • Nai Zhou
  • Jian Zhao
  • Yanan Zhang

Joint entity and relation extraction (RE) construct a framework for unifying entity recognition and relationship extraction, and the approach can exploit the dependencies between the two tasks to improve the performance of the task. However, the existing tasks still have the following two problems. First, when the model extracts entity information, the boundary is blurred. Secondly, there are mostly implicit interactions between modules, that is, the interactive information is hidden inside the model, and the implicit interactions are often insufficient in the degree of interaction and lack of interpretability. To this end, this study proposes a joint entity and relation extraction model (ESEI) based on E fficient S ampling and E xplicit I nteraction. We innovatively divide negative samples into sentences based on whether they overlap with positive samples, which improves the model’s ability to extract entity word boundary information by controlling the sampling ratio. In order to increase the explicit interaction ability between the models, we introduce a heterogeneous graph neural network (GNN) into the model, which will serve as a bridge linking the entity recognition module and the relation extraction module, and enhance the interaction between the modules through information transfer. Our method substantially improves the model’s discriminative power on entity extraction tasks and enhances the interaction between relation extraction tasks and entity extraction tasks. Experiments show that the method is effective, we validate our method on four datasets, and for joint entity and relation extraction, our model improves the F1 score on multiple datasets.

Details DOI

AAAI Conference 2023 Conference Paper

Contrastive Open Set Recognition

  • Baile Xu
  • Furao Shen
  • Jian Zhao

In conventional recognition tasks, models are only trained to recognize learned targets, but it is usually difficult to collect training examples of all potential categories. In the testing phase, when models receive test samples from unknown classes, they mistakenly classify the samples into known classes. Open set recognition (OSR) is a more realistic recognition task, which requires the classifier to detect unknown test samples while keeping a high classification accuracy of known classes. In this paper, we study how to improve the OSR performance of deep neural networks from the perspective of representation learning. We employ supervised contrastive learning to improve the quality of feature representations, propose a new supervised contrastive learning method that enables the model to learn from soft training targets, and design an OSR framework on its basis. With the proposed method, we are able to make use of label smoothing and mixup when training deep neural networks contrastively, so as to improve both the robustness of outlier detection in OSR tasks and the accuracy in conventional classification tasks. We validate our method on multiple benchmark datasets and testing scenarios, achieving experimental results that verify the effectiveness of the proposed method.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

DIFFER:Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning

  • Xunhan Hu
  • Jian Zhao
  • Wengang Zhou
  • Ruili Feng
  • Houqiang Li

Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting to achieve more muscular efficiency and fairness when diversity is observed. Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness. Code is available in supplement material.

PDF Details

JAIR Journal 2022 Journal Article

C-Face: Using Compare Face on Face Hallucination for Low-Resolution Face Recognition

  • Feng Han
  • Xudong Wang
  • Furao Shen
  • Jian Zhao

Face hallucination is a task of generating high-resolution (HR) face images from low-resolution (LR) inputs, which is a subfield of the general image super-resolution. However, most of the previous methods only consider the visual effect, ignoring how to maintain the identity of the face. In this work, we propose a novel face hallucination model, called C-Face network, which can generate HR images with high visual quality while preserving the identity information. A face recognition network is used to extract the identity features in the training process. In order to make the reconstructed face images keep the identity information to a great extent, a novel metric, i.e., C-Face loss, is proposed. We also propose a new training algorithm to deal with the convergence problem. Moreover, since our work mainly focuses on the recognition accuracy of the output, we integrate face recognition into the face hallucination process which ensures that the model can be used in real scenarios. Extensive experiments on two large scale face datasets demonstrate that our C-Face network has the best performance compared with other state-of-the-art methods.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning

  • Mingyu Yang
  • Jian Zhao
  • Xunhan Hu
  • Wengang Zhou
  • Jiangcheng Zhu
  • Houqiang Li

Cooperative multi-agent reinforcement learning (MARL) has made prominent progress in recent years. For training efficiency and scalability, most of the MARL algorithms make all agents share the same policy or value network. However, in many complex multi-agent tasks, different agents are expected to possess specific abilities to handle different subtasks. In those scenarios, sharing parameters indiscriminately may lead to similar behavior across all agents, which will limit the exploration efficiency and degrade the final performance. To balance the training complexity and the diversity of agent behavior, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL. Specifically, we first introduce a subtask encoder to construct a vector representation for each subtask according to its identity. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy, which can dynamically group agents with similar abilities into the same subtask. In this way, agents dealing with the same subtask share their learning of specific abilities and different subtasks correspond to different specific abilities. We further introduce two regularizers to increase the representation difference between subtasks and stabilize the training by discouraging agents from frequently changing subtasks, respectively. Empirical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark and Google Research Football.

PDF Details

IJCAI Conference 2022 Conference Paper

Semantic Compression Embedding for Generative Zero-Shot Learning

  • Ziming Hong
  • Shiming Chen
  • Guo-Sen Xie
  • Wenhan Yang
  • Jian Zhao
  • Yuanjie Shao
  • Qinmu Peng
  • Xinge You

Generative methods have been successfully applied in zero-shot learning (ZSL) by learning an implicit mapping to alleviate the visual-semantic domain gaps and synthesizing unseen samples to handle the data imbalance between seen and unseen classes. However, existing generative methods simply use visual features extracted by the pre-trained CNN backbone. These visual features lack attribute-level semantic information. Consequently, seen classes are indistinguishable, and the knowledge transfer from seen to unseen classes is limited. To tackle this issue, we propose a novel Semantic Compression Embedding Guided Generation (SC-EGG) model, which cascades a semantic compression embedding network (SCEN) and an embedding guided generative network (EGGN). The SCEN extracts a group of attribute-level local features for each sample and further compresses them into the new low-dimension visual feature. Thus, a dense-semantic visual space is obtained. The EGGN learns a mapping from the class-level semantic space to the dense-semantic visual space, thus improving the discriminability of the synthesized dense-semantic unseen visual features. Extensive experiments on three benchmark datasets, i. e. , CUB, SUN and AWA2, demonstrate the significant performance gains of SC-EGG over current state-of-the-art methods and its baselines.

PDF Details DOI

EAAI Journal 2022 Journal Article

Waveform level adversarial example generation for joint attacks against both automatic speaker verification and spoofing countermeasures

  • Xingyu Zhang
  • Xiongwei Zhang
  • Wei Liu
  • Xia Zou
  • Meng Sun
  • Jian Zhao

Adversarial examples crafted to deceive Automatic Speaker Verification (ASV) systems have attracted a lot of attention when studying the vulnerability of ASV. However, real-world ASV systems usually work together with spoofing countermeasures (CM) to exclude fake voices generated by text-to-speech (TTS) or voice conversion (VC). The deployment of CM would reduce the capability of the adversarial samples on deceiving ASV. Although additional perturbations against CM may be generated and put on the crafted adversarial examples against ASV to yield new adversarial examples against both ASV and CM, those additional perturbations would however hinder the examples’ adversarial effectiveness on ASV. In this paper, a novel joint approach is proposed to generate adversarial examples by considering attacking ASV and CM simultaneously. For any voice from TTS, VC or a real-world speaker, our crafted adversarial perturbations will turn its original labels on CM and speaker ID to bonafide and some target speaker ID, correspondingly. In our approach, a differentiable front-end is introduced to replace the conventional hand-crafted time–frequency feature extractor. Perturbations can thus be estimated by updating the gradients of the joint objective of ASV and CM on the waveform variables. The proposed method has demonstrated a 99. 3% success rate on white-box logical access attacks to deceive ASV and CM simultaneously, which outperforms the baselines of 65. 3% and 36. 7%. Furthermore, transferability on black-box and physical settings has also been validated.

Details DOI

TIST Journal 2020 Journal Article

Learning Generalizable and Identity-Discriminative Representations for Face Anti-Spoofing

  • Xiaoguang Tu
  • Zheng Ma
  • Jian Zhao
  • Guodong DU
  • Mei Xie
  • Jiashi Feng

Face anti-spoofing aims to detect presentation attack to face recognition--based authentication systems. It has drawn growing attention due to the high security demand. The widely adopted CNN-based methods usually well recognize the spoofing faces when training and testing spoofing samples display similar patterns, but their performance would drop drastically on testing spoofing faces of novel patterns or unseen scenes, leading to poor generalization performance. Furthermore, almost all current methods treat face anti-spoofing as a prior step to face recognition, which prolongs the response time and makes face authentication inefficient. In this article, we try to boost the generalizability and applicability of face anti-spoofing methods by designing a new generalizable face authentication CNN (GFA-CNN) model with three novelties. First, GFA-CNN introduces a simple yet effective total pairwise confusion loss for CNN training that properly balances contributions of all spoofing patterns for recognizing the spoofing faces. Second, it incorporate a fast domain adaptation component to alleviate negative effects brought by domain variation. Third, it deploys filter diversification learning to make the learned representations more adaptable to new scenes. In addition, the proposed GFA-CNN works in a multi-task manner—it performs face anti-spoofing and face recognition simultaneously. Experimental results on five popular face anti-spoofing and face recognition benchmarks show that GFA-CNN outperforms previous face anti-spoofing methods on cross-test protocols significantly and also well preserves the identity information of input face images.

Details DOI

AAAI Conference 2019 Conference Paper

Look across Elapse: Disentangled Representation Learning and Photorealistic Cross-Age Face Synthesis for Age-Invariant Face Recognition

  • Jian Zhao
  • Yu Cheng
  • Yi Cheng
  • Yang Yang
  • Fang Zhao
  • Jianshu Li
  • Hengzhu Liu
  • Shuicheng Yan

Despite the remarkable progress in face recognition related technologies, reliably recognizing faces across ages still remains a big challenge. The appearance of a human face changes substantially over time, resulting in significant intraclass variations. As opposed to current techniques for ageinvariant face recognition, which either directly extract ageinvariant features for recognition, or first synthesize a face that matches target age before feature extraction, we argue that it is more desirable to perform both tasks jointly so that they can leverage each other. To this end, we propose a deep Age-Invariant Model (AIM) for face recognition in the wild with three distinct novelties. First, AIM presents a novel unified deep architecture jointly performing cross-age face synthesis and recognition in a mutual boosting way. Second, AIM achieves continuous face rejuvenation/aging with remarkable photorealistic and identity-preserving properties, avoiding the requirement of paired data and the true age of testing samples. Third, we develop effective and novel training strategies for end-to-end learning the whole deep architecture, which generates powerful age-invariant face representations explicitly disentangled from the age variation. Extensive experiments on several cross-age datasets (MORPH, CACD and FG-NET) demonstrate the superiority of the proposed AIM model over the state-of-the-arts. Benchmarking our model on one of the most popular unconstrained face recognition datasets IJB-C additionally verifies the promising generalizability of AIM in recognizing faces in the wild.

PDF Details

IJCAI Conference 2019 Conference Paper

Multi-Prototype Networks for Unconstrained Set-based Face Recognition

  • Jian Zhao
  • Jianshu Li
  • Xiaoguang Tu
  • Fang Zhao
  • Yuan Xin
  • Junliang Xing
  • Hengzhu Liu
  • Shuicheng Yan

In this paper, we address the challenging unconstrained set-based face recognition problem where each subject face is instantiated by a set of media (images and videos) instead of a single image. Naively aggregating information from all the media within a set would suffer from the large intra-set variance caused by heterogeneous factors (e. g. , varying media modalities, poses and illumination) and fail to learn discriminative face representations. A novel Multi-Prototype Network (MP- Net) model is thus proposed to learn multiple prototype face representations adaptively from the media sets. Each learned prototype is representative for the subject face under certain condition in terms of pose, illumination and media modality. Instead of handcrafting the set partition for prototype learn- ing, MPNet introduces a Dense SubGraph (DSG) learning sub-net that implicitly untangles inconsistent media and learns a number of representative prototypes. Qualitative and quantitative experiments clearly demonstrate the superiority of the proposed model over state-of-the-arts.

PDF Details

IJCAI Conference 2018 Conference Paper

3D-Aided Deep Pose-Invariant Face Recognition

  • Jian Zhao
  • Lin Xiong
  • Yu Cheng
  • Yi Cheng
  • Jianshu Li
  • Li Zhou
  • Yan Xu
  • Jayashree Karlekar

Learning from synthetic faces, though perhaps appealing for high data efficiency, may not bring satisfactory performance due to the distribution discrepancy of the synthetic and real face images. To mitigate this gap, we propose a 3D-Aided Deep Pose-Invariant Face Recognition Model (3D-PIM), which automatically recovers realistic frontal faces from arbitrary poses through a 3D face model in a novel way. Specifically, 3D-PIM incorporates a simulator with the aid of a 3D Morphable Model (3D MM) to obtain shape and appearance prior for accelerating face normalization learning, requiring less training data. It further leverages a global-local Generative Adversarial Network (GAN) with multiple critical improvements as a refiner to enhance the realism of both global structures and local details of the face simulator’s output using unlabelled real data only, while preserving the identity information. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks clearly demonstrate superiority of the proposed model over state-of-the-arts.

PDF Details

NeurIPS Conference 2017 Conference Paper

Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis

  • Jian Zhao
  • Lin Xiong
  • Panasonic Karlekar Jayashree
  • Jianshu Li
  • Fang Zhao
  • Zhecan Wang
  • Panasonic Sugiri Pranata
  • Panasonic Shengmei Shen

Synthesizing realistic profile faces is promising for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by populating samples with extreme poses and avoiding tedious annotations. However, learning from synthetic faces may not achieve the desired performance due to the discrepancy between distributions of the synthetic and real face images. To narrow this gap, we propose a Dual-Agent Generative Adversarial Network (DA-GAN) model, which can improve the realism of a face simulator's output using unlabeled real faces, while preserving the identity information during the realism refinement. The dual agents are specifically designed for distinguishing real v. s. fake and identities simultaneously. In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses. DA-GAN leverages a fully convolutional network as the generator to generate high-resolution images and an auto-encoder as the discriminator with the dual agents. Besides the novel architecture, we make several key modifications to the standard GAN to preserve pose and texture, preserve identity and stabilize training process: (i) a pose perception loss; (ii) an identity perception loss; (iii) an adversarial loss with a boundary equilibrium regularization term. Experimental results show that DA-GAN not only presents compelling perceptual results but also significantly outperforms state-of-the-arts on the large-scale and challenging NIST IJB-A unconstrained face recognition benchmark. In addition, the proposed DA-GAN is also promising as a new approach for solving generic transfer learning problems more effectively.

PDF Details
v2026.09.13