Arrow Research search

Author name cluster

Chen Qian

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

36 papers
2 author rows

Possible papers

36

EAAI Journal 2026 Journal Article

Cooperative decision-making of unmanned aerial vehicles: A multi-agent reinforcement learning approach

  • Ziyi Wang
  • Guoliang Ma
  • Jian Guo
  • Chen Qian
  • Yang Gao
  • Zhuo Huang

Cooperative decision-making of unmanned aerial vehicles (UAVs) for military missions is a crucial research topic. However, the ability constraints of heterogeneous UAVs in real-world scenarios bring significant challenges to the cooperative decision-making process. To address these issues, this paper proposes a multi-agent proximal policy optimization (MAPPO) algorithm with a flexible observation feature encoding (FOFE) mechanism and a Mamba-based memory structure. Firstly, the cooperative decision-making problem for reconnaissance-strike integrated fixed-wing UAV (RSUAV) swarms is formulated as a distributed partially observable Markov decision process (Dec-POMDP). Secondly, to address the variability and incompleteness in observation inputs, an FOFE strategy is introduced. This allows the network to process multi-channel and variable-length data effectively. Furthermore, the Mamba model is incorporated to capture temporal dependencies in historical observations. This enhances decision-making in prolonged missions. Under this multi-agent reinforcement learning (MARL) framework, each RSUAV can make autonomous decisions in a decentralized manner. The simulation results show that the proposed algorithm improves the completion ratio ( > 10. 1%), survival ratio ( > 14. 8%), and reduces completion time ( > 21. 5%) compared to baselines. It also exhibits strong generalization capability and holds practical feasibility for deployment on edge computing devices. Therefore, this approach enables effective cooperative decision-making under the ability constraints of heterogeneous RSUAVs.

Details DOI

IROS Conference 2025 Conference Paper

Abdominal Undulation with Compliant Mechanism Improves Flight Performance of a Biomimetic Robotic Butterfly

  • Xuyi Lian
  • Mingyu Luo
  • Te Lin
  • Chen Qian
  • Tiefeng Li

This paper presents the design, modeling, and experimental validation of a biomimetic robotic butterfly (BRB) that integrates a compliant mechanism to achieve coupled wing-abdomen motion. Drawing inspiration from the natural flight dynamics of butterflies, a theoretical model is developed to investigate the impact of abdominal undulation on flight performance. To validate the model, motion capture experiments are conducted on three configurations: a BRB without an abdomen, with a fixed abdomen, and with an undulating abdomen. The results demonstrate that abdominal undulation enhances lift generation, extends flight duration, and stabilizes pitch oscillations, thereby improving overall flight performance. These findings underscore the significance of wing-abdomen interaction in flapping-wing aerial vehicles (FWAVs) and lay the groundwork for future advancements in energy-efficient biomimetic flight designs.

Details

AAAI Conference 2025 Conference Paper

AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks

  • Zekang Yang
  • Wang Zeng
  • Sheng Jin
  • Chen Qian
  • Ping Luo
  • Wentao Liu

Automated machine learning (AutoML) is a collection of techniques designed to automate the machine learning development process. While traditional AutoML approaches have been successfully applied in several critical steps of model development (e.g. hyperparameter optimization), there lacks a AutoML system that automates the entire end-to-end model production workflow for computer vision. To fill this blank, we propose a novel request-to-model task, which involves understanding the user's natural language request and execute the entire workflow to output production-ready models. This empowers non-expert individuals to easily build task-specific models via a user-friendly language interface. To facilitate development and evaluation, we develop a new experimental platform called AutoMMLab and a new benchmark called LAMP for studying key components in the end-to-end request-to-model pipeline. Hyperparameter optimization (HPO) is one of the most important components for AutoML. Traditional approaches mostly rely on trial-and-error, leading to inefficient parameter search. To solve this problem, we propose a novel LLM-based HPO algorithm, called HPO-LLaMA. Equipped with extensive knowledge and experience in model hyperparameter tuning, HPO-LLaMA achieves significant improvement of HPO efficiency.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning

  • Chen Qian
  • Dongrui Liu
  • Haochen Wen
  • Zhen Bai
  • Yong Liu
  • Jing Shao

Large reasoning models (LRMs) have demonstrated impressive capabilities in complex problem-solving, yet their internal reasoning mechanisms remain poorly understood. In this paper, we investigate the reasoning trajectories of LRMs from an information-theoretic perspective. By tracking how mutual information (MI) between intermediate representations and the correct answer evolves during LRM reasoning, we observe an interesting MI peaks phenomenon: the MI at specific generative steps exhibits a sudden and significant increase during LRM's reasoning process. We theoretically analyze such phenomenon and show that as MI increases, the probability of model's prediction error decreases. Furthermore, these MI peaks often correspond to tokens expressing reflection or transition, such as "Hmm", "Wait" and "Therefore, " which we term as the thinking tokens. We then demonstrate that these thinking tokens are crucial for LRM's reasoning performance, while other tokens has minimal impacts. Building on these analyses, we propose two simple yet effective methods to improve LRM's reasoning performance, by delicately leveraging these thinking tokens. Overall, our work provides novel insights into the reasoning mechanisms of LRMs and offers practical ways to improve their reasoning capabilities. The code is available at \url{https: //github. com/ChnQ/MI-Peaks}.

PDF Details

ICLR Conference 2025 Conference Paper

Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence

  • Weize Chen
  • Ziming You
  • Ran Li
  • Yitong Guan
  • Chen Qian
  • Chenyang Zhao
  • Cheng Yang 0002
  • Ruobing Xie

The rapid advancement of large language models (LLMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often struggle with integrating diverse capable third-party agents due to reliance on agents defined within their own ecosystems. They also face challenges in simulating distributed environments, as most frameworks are limited to single-device setups. Furthermore, these frameworks often rely on hard-coded communication pipelines, limiting their adaptability to dynamic task requirements. Inspired by the concept of the Internet, we propose the Internet of Agents (IoA), a novel framework that addresses these limitations by providing a flexible and scalable platform for LLM-based multi-agent collaboration. IoA introduces an agent integration protocol, an instant-messaging-like architecture design, and dynamic mechanisms for agent teaming and conversation flow control. Through extensive experiments on general assistant tasks, embodied AI tasks, and retrieval-augmented generation benchmarks, we demonstrate that IoA consistently outperforms state-of-the-art baselines, showcasing its ability to facilitate effective collaboration among heterogeneous agents. IoA represents a step towards linking diverse agents in an Internet-like environment, where agents can seamlessly collaborate to achieve greater intelligence and capabilities. We will release our code to facilitate further research.

Details

NeurIPS Conference 2025 Conference Paper

Multi-Agent Collaboration via Evolving Orchestration

  • Yufan Dang
  • Chen Qian
  • Xueheng Luo
  • Jingru Fan
  • Zihao Xie
  • Ruijie Shi
  • Weize Chen
  • Cheng Yang

Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving. While recent research explores multi-agent collaboration among LLMs, most approaches rely on static organizational structures that struggle to adapt as task complexity and agent numbers grow, resulting in coordination overhead and inefficiencies. To this end, we propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states. This orchestrator is trained via reinforcement learning to adaptively sequence and prioritize agents, enabling flexible and evolvable collective reasoning. Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs. Analyses further reveal that the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures under the orchestrator’s evolution. Our code is available at https: //github. com/OpenBMB/ChatDev/tree/puppeteer.

PDF Details

ICLR Conference 2025 Conference Paper

Scaling Large Language Model-based Multi-Agent Collaboration

  • Chen Qian
  • Zihao Xie
  • Yifei Wang
  • Wei Liu 0161
  • Kunlun Zhu
  • Hanchen Xia
  • Yufan Dang
  • Zhuoyun Du

Recent breakthroughs in large language model-driven autonomous agents have revealed that multi-agent collaboration often surpasses each individual through collective reasoning. Inspired by the neural scaling law—increasing neurons enhances performance, this study explores whether the continuous addition of collaborative agents can yield similar benefits. Technically, we utilize directed acyclic graphs to organize agents into a multi-agent collaboration network (MacNet), upon which their interactive reasoning is topologically orchestrated for autonomous task solving. Extensive evaluations reveal that it effectively supports collaboration among over a thousand agents, with irregular topologies outperforming regular ones. We also identify a collaborative scaling law—the overall performance follows a logistic growth pattern as agents scale, with collaborative emergence occurring earlier than traditional neural emergence. We speculate this may be because scaling agents catalyzes their multidimensional considerations during interactive reflection and refinement, thereby producing more comprehensive artifacts. The code is available at https://github.com/OpenBMB/ChatDev/tree/macnet.

Details

AAAI Conference 2025 Conference Paper

ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries

  • Wangyu Xue
  • Chen Qian
  • Jiayi Wu
  • Yang Zhou
  • Wentao Liu
  • Ju Ren
  • Siming Fan
  • Yaoxue Zhang

Existing research on human-centric video understanding typically focuses on analyzing specific moments or entire videos. However, many applications require higher precision at the frame level. In this work, we propose a novel task, BestShot, which aims to locate highlight frames within human-centric videos through language queries. This task requires not only a deep semantic understanding of human actions but also precise temporal localization. To support this task, we introduce the BestShot Benchmark. The benchmark is meticulously constructed by combining human-annotated highlight frames, duration labels and detailed textual descriptions. These descriptions cover three critical elements: (1) Visual content; (2) Fine-grained actions; and (3) Human pose descriptions. Together, these elements provide the necessary precision to identify the exact highlight frames in videos. To tackle this problem, we have collected two distinct datasets: (i) ShotGPT4o Dataset, which is algorithmically generated by GPT-4o and (ii) Image-SMPLText Dataset, which features large-scale and accurate per-frame pose descriptions using PoseScript and existing pose estimation datasets. Based on these datasets, we present a strong baseline model, ShotVL, fine-tuned from InternVL, specifically for BestShot. We highlight the impressive zero-shot capabilities of our model and offer comparative analyses with existing state-of-the-art (SOTA) models. ShotVL demonstrates a significant 64% improvement over InternVL on the BestShot Benchmark and a notable 68% improvement on the THUMOS14 Benchmark, while maintaining SOTA performance in general image classification and retrieval.

PDF Details DOI

JBHI Journal 2024 Journal Article

A Faithful Deep Sensitivity Estimation for Accelerated Magnetic Resonance Imaging

  • Zi Wang
  • Haoming Fang
  • Chen Qian
  • Boxuan Shi
  • Lijun Bao
  • Liuhong Zhu
  • Jianjun Zhou
  • Wenping Wei

Magnetic resonance imaging (MRI) is an essential diagnostic tool that suffers from prolonged scan time. To alleviate this limitation, advanced fast MRI technology attracts extensive research interests. Recent deep learning has shown its great potential in improving image quality and reconstruction speed. Faithful coil sensitivity estimation is vital for MRI reconstruction. However, most deep learning methods still rely on pre-estimated sensitivity maps and ignore their inaccuracy, resulting in the significant quality degradation of reconstructed images. In this work, we propose a Joint Deep Sensitivity estimation and Image reconstruction network, called JDSI. During the image artifacts removal, it gradually provides more faithful sensitivity maps with high-frequency information, leading to improved image reconstructions. To understand the behavior of the network, the mutual promotion of sensitivity estimation and image reconstruction is revealed through the visualization of network intermediate results. Results on in vivo datasets and radiologist reader study demonstrate that, for both calibration-based and calibrationless reconstruction, the proposed JDSI achieves the state-of-the-art performance visually and quantitatively, especially when the acceleration factor is high. Additionally, JDSI owns nice robustness to patients and autocalibration signals.

Details DOI

ICLR Conference 2024 Conference Paper

AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors

  • Weize Chen
  • Yusheng Su
  • Jingwei Zuo
  • Cheng Yang 0002
  • Chenfei Yuan
  • Chi-Min Chan
  • Heyang Yu
  • Yaxi Lu

Autonomous agents empowered by Large Language Models (LLMs) have undergone significant improvements, enabling them to generalize across a broad spectrum of tasks. However, in real-world scenarios, cooperation among individuals is often required to enhance the efficiency and effectiveness of task accomplishment. Hence, inspired by human group dynamics, we propose a multi-agent framework AgentVerse that can effectively orchestrate a collaborative group of expert agents as a greater-than-the-sum-of-its-parts system. Our experiments demonstrate that AgentVerse can proficiently deploy multi-agent groups that outperform a single agent. Extensive experiments on text understanding, reasoning, coding, tool utilization, and embodied AI confirm the effectiveness of AgentVerse. Moreover, our analysis of agent interactions within AgentVerse reveals the emergence of specific collaborative behaviors, contributing to heightened group efficiency. We will release our codebase, AgentVerse, to further facilitate multi-agent research.

Details

EAAI Journal 2024 Journal Article

An adaptive bidirectional quick optimal Rapidly-exploring Random Tree algorithm for path planning

  • Zhuo Huang
  • Yang Gao
  • Jian Guo
  • Chen Qian
  • Qingwei Chen

In recent decades, sampling-based algorithms have played an important role in path planning. As an extended algorithm of Rapidly-exploring Random Tree(RRT), the optimal RRT(RRT*) has advantages in asymptotic optimality. However, it still suffers from a slow convergence rate and expensive initial solution heavily. Besides, it cannot provide an effective path planning solution when dealing with narrow passages. To overcome this problem, an adaptive bidirectional quick RRT*(ABQ-RRT*) method is proposed for Unmanned Aerial Vehicle(UAV) in narrow passages. Firstly, the cost function has been redefined by taking into account both the Euclidean distance and the turning angle of the UAV. Secondly, the number of failures in the collision detection is considered and an adaptive goal-biased sampling strategy is adopted to obtain higher-quality sampling points. Thirdly, by local sampling near the obstacles, the type of the local environment is quickly judged. Also, the repulsion field is superimposed to effectively avoid obstacles. Then, during the process of selecting the optimal parent node from the set of candidate nodes, the sorted possible parents consider both the neighborhood nodes and their ancestors to obtain better paths. At last, the effectiveness of ABQ-RRT* is confirmed by comparing it with several other improved RRT algorithms through simulations.

Details DOI

NeurIPS Conference 2024 Conference Paper

Autonomous Agents for Collaborative Task under Information Asymmetry

  • Wei Liu
  • Chenxi Wang
  • Yifei Wang
  • Zihao Xie
  • Rennai Qiu
  • Yufan Dang
  • Zhuoyun Du
  • Weize Chen

Large Language Model Multi-Agent Systems (LLM-MAS) have greatly progressed in solving complex tasks. It communicates among agents within the system to collaboratively solve tasks, under the premise of shared information. However, when agents' collaborations are leveraged to perform multi-person tasks, a new challenge arises due to information asymmetry, since each agent can only access the information of its human user. Previous MAS struggle to complete tasks under this condition. To address this, we propose a new MAS paradigm termed iAgents, which denotes Informative Multi-Agent Systems. In iAgents, the human social network is mirrored in the agent network, where agents proactively exchange human information necessary for task resolution, thereby overcoming information asymmetry. iAgents employs a novel agent reasoning mechanism, InfoNav, to navigate agents' communication towards effective information exchange. Together with InfoNav, iAgents organizes human information in a mixed memory to provide agents with accurate and comprehensive information for exchange. Additionally, we introduce InformativeBench, the first benchmark tailored for evaluating LLM agents' task-solving ability under information asymmetry. Experimental results show that iAgents can collaborate within a social network of 140 individuals and 588 relationships, autonomously communicate over 30 turns, and retrieve information from nearly 70, 000 messages to complete tasks within 3 minutes.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models

  • Xin Li
  • Weize Chen
  • Qizhi Chu
  • Haopeng Li
  • Zhaojun Sun
  • Ran Li
  • Chen Qian
  • Yiwei Wei

The need to analyze graphs is ubiquitous across various fields, from social networks to biological research and recommendation systems. Therefore, enabling the ability of large language models (LLMs) to process graphs is an important step toward more advanced general intelligence. However, current LLM benchmarks on graph analysis require models to directly reason over the prompts describing graphtopology, and are thus limited to small graphs with only a few dozens of nodes. In contrast, human experts typically write programs based on popular libraries for task solving, and can thus handle graphs with different scales. To this end, a question naturally arises: can LLMs analyze graphs like professionals? In this paper, we introduce ProGraph, a manually crafted benchmark containing 3 categories of graph tasks. The benchmark expects solutions based on programming instead of directly reasoning over raw inputs. Our findings reveal that the performance of current LLMs is unsatisfactory, with the best model achieving only 36% accuracy. To bridge this gap, we propose LLM4Graph datasets, which include crawled documents and auto-generated codes based on 6 widely used graph libraries. By augmenting closed-source LLMs with document retrieval and fine-tuning open-source ones on the codes, we show 11-32% absolute improvements in their accuracies. Our results underscore that the capabilities of LLMs in handling structured data are still under-explored, and show the effectiveness of LLM4Graph in enhancing LLMs’ proficiency of graph analysis. The benchmark, datasets and enhanced open-sourcemodels are available at https: //github. com/BUPT-GAMMA/ProGraph.

PDF Details DOI

IROS Conference 2024 Conference Paper

Effects of fiber number and density on fiber jamming: Towards follow-the-leader deployment of a continuum robot

  • Chen Qian
  • Tangyou Liu
  • Liao Wu

Fiber jamming modules (FJMs) offer flexibility and quick stiffness variation, making them suitable for followthe-leader (FTL) motions in continuum robots, which is ideal for minimally invasive surgery (MIS). However, their potential has not been fully exploited, particularly in designing and manufacturing small-sized FJMs with high stiffness variation. Although existing research has focused on factors like fiber materials and geometry to maximize stiffness variation, the results often do not apply to FJMs for MIS due to size constraints. Meanwhile, other factors such as fiber number and packing density, less significant to large FJMs but critical to smallsized FJMs, have received insufficient investigation regarding their impact on the stiffness variation for FTL deployment. In this paper, we design and fabricate FJMs with a diameter of 4mm. Through theoretical and experimental analysis, we find that fiber number and packing density significantly affect both absolute stiffness and stiffness variation. Our experiments confirm the feasibility of using FJMs in a medical FTL robot design. The optimal configuration is a 4mm FJM with 0. 4mm fibers at a 56% packing density, achieving up to 3400% stiffness variation. A video demonstration of a prototype robot using the suggested parameters for achieving FTL motions can be found at https://youtu.be/7pI5U0z7kcE.

Details

NeurIPS Conference 2024 Conference Paper

Fairness without Harm: An Influence-Guided Active Sampling Approach

  • Jinlong Pang
  • Jialu Wang
  • Zhaowei Zhu
  • Yuanshun Yao
  • Chen Qian
  • Yang Liu

The pursuit of fairness in machine learning (ML), ensuring that the models do not exhibit biases toward protected demographic groups, typically results in a compromise scenario. This compromise can be explained by a Pareto frontier where given certain resources (e. g. , data), reducing the fairness violations often comes at the cost of lowering the model accuracy. In this work, we aim to train models that mitigate group fairness disparity without causing harm to model accuracy. Intuitively, acquiring more data is a natural and promising approach to achieve this goal by reaching a better Pareto frontier of the fairness-accuracy tradeoff. The current data acquisition methods, such as fair active learning approaches, typically require annotating sensitive attributes. However, these sensitive attribute annotations should be protected due to privacy and safety concerns. In this paper, we propose a tractable active data sampling algorithm that does not rely on training group annotations, instead only requiring group annotations on a small validation set. Specifically, the algorithm first scores each new example by its influence on fairness and accuracy evaluated on the validation dataset, and then selects a certain number of examples for training. We theoretically analyze how acquiring more data can improve fairness without causing harm, and validate the possibility of our sampling approach in the context of risk disparity. We also provide the upper bound of generalization error and risk disparity as well as the corresponding connections. Extensive experiments on real-world data demonstrate the effectiveness of our proposed algorithm. Our code is available at github. com/UCSC-REAL/FairnessWithoutHarm.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

From Transparent to Opaque: Rethinking Neural Implicit Surfaces with $\alpha$-NeuS

  • Haoran Zhang
  • Junkai Deng
  • Xuhui Chen
  • Fei Hou
  • Wencheng Wang
  • Hong Qin
  • Chen Qian
  • Ying He

Traditional 3D shape reconstruction techniques from multi-view images, such as structure from motion and multi-view stereo, face challenges in reconstructing transparent objects. Recent advances in neural radiance fields and its variants primarily address opaque or transparent objects, encountering difficulties to reconstruct both transparent and opaque objects simultaneously. This paper introduces $\alpha$-NeuS$\textemdash$an extension of NeuS$\textemdash$that proves NeuS is unbiased for materials from fully transparent to fully opaque. We find that transparent and opaque surfaces align with the non-negative local minima and the zero iso-surface, respectively, in the learned distance field of NeuS. Traditional iso-surfacing extraction algorithms, such as marching cubes, which rely on fixed iso-values, are ill-suited for such data. We develop a method to extract the transparent and opaque surface simultaneously based on DCUDF. To validate our approach, we construct a benchmark that includes both real-world and synthetic scenes, demonstrating its practical utility and effectiveness. Our data and code are publicly available at https: //github. com/728388808/alpha-NeuS.

PDF Details DOI

JMLR Journal 2024 Journal Article

KerasCV and KerasNLP: Multi-framework Models

  • Matthew Watson
  • Divyashree Shivakumar Sreepathihalli
  • François Chollet
  • Martin Górner
  • Kiranbir Sodhia
  • Ramesh Sampath
  • Tirth Patel
  • Haifeng Jin

We present the Keras domain packages KerasCV and KerasNLP, extensions of the Keras API for Computer Vision and Natural Language Processing workflows, capable of running on either JAX, TensorFlow, or PyTorch. These domain packages are designed to enable fast experimentation, with a focus on ease-of-use and performance. We adopt a modular, layered design: at the library's lowest level of abstraction, we provide building blocks for creating models and data preprocessing pipelines, and at the library's highest level of abstraction, we provide pretrained "task" models for popular architectures such as Stable Diffusion, YOLOv8, GPT2, BERT, Mistral, CLIP, Gemma, T5, etc. Task models have built-in preprocessing, pretrained weights, and can be fine-tuned on raw inputs. To enable efficient training, we support XLA compilation for all models, and run all preprocessing via a compiled graph of TensorFlow operations using the tf.data API. The libraries are fully open-source (Apache 2.0 license) and available on GitHub. Keywords: KerasCV, KerasNLP, Keras multi-backend, Deep learning, Generative AI [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details

NeurIPS Conference 2024 Conference Paper

KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

  • Jie Yang
  • Wang Zeng
  • Sheng Jin
  • Lumin Xu
  • Wentao Liu
  • Chen Qian
  • Ruimao Zhang

Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e. g. , the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Amodal Instance Segmentation via Prior-Guided Expansion

  • Junjie Chen
  • Li Niu
  • Jianfu Zhang
  • Jianlou Si
  • Chen Qian
  • Liqing Zhang

Amodal instance segmentation aims to infer the amodal mask, including both the visible part and occluded part of each object instance. Predicting the occluded parts is challenging. Existing methods often produce incomplete amodal boxes and amodal masks, probably due to lacking visual evidences to expand the boxes and masks. To this end, we propose a prior-guided expansion framework, which builds on a two-stage segmentation model (i.e., Mask R-CNN) and performs box-level (resp., pixel-level) expansion for amodal box (resp., mask) prediction, by retrieving regression (resp., flow) transformations from a memory bank of expansion prior. We conduct extensive experiments on KINS, D2SA, and COCOA cls datasets, which show the effectiveness of our method.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Knowledge Diffusion for Distillation

  • Tao Huang
  • Yuan Zhang
  • Mingkai Zheng
  • Shan You
  • Fei Wang
  • Chen Qian
  • Chang Xu

The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature, and propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher features. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https: //github. com/hunto/DiffKD.

PDF Details

NeurIPS Conference 2023 Conference Paper

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

  • Dongwei Pan
  • Long Zhuo
  • Jingtan Piao
  • Huiwen Luo
  • Wei Cheng
  • Yuxin Wang
  • Siming Fan
  • Shengqi Liu

Synthesizing high-fidelity head avatars is a central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is the inadequate datasets -- 1) current public datasets can only support researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume, and narrow distribution over different attributes, such as expressions, ages, and accessories. In this paper, we present RenderMe-360, a comprehensive 4D human head dataset to drive advance in head avatar algorithms across different scenarios. It contains massive data assets, with 243+ million complete head frames and over 800k video sequences from 500 different identities captured by multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured in 360 degrees via 60 synchronized, high-resolution 2K cameras. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked to perform various dynamic motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: the dataset provides annotations with different granularities: cameras' parameters, background matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description. Based on the dataset, we build a comprehensive benchmark for head avatar research, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and flaws of state-of-the-art methods. RenderMe-360 opens the door for future exploration in modern head avatars. All of the data, code, and models will be publicly available at https: //renderme-360. github. io/.

PDF Details

NeurIPS Conference 2022 Conference Paper

Green Hierarchical Vision Transformer for Masked Image Modeling

  • Lang Huang
  • Shan You
  • Mingkai Zheng
  • Fei Wang
  • Chen Qian
  • Toshihiko Yamasaki

We present an efficient approach for Masked Image Modeling (MIM) with hierarchical Vision Transformers (ViTs), allowing the hierarchical ViTs to discard masked patches and operate only on the visible ones. Our approach consists of three key designs. First, for window attention, we propose a Group Window Attention scheme following the Divide-and-Conquer strategy. To mitigate the quadratic complexity of the self-attention w. r. t. the number of patches, group attention encourages a uniform partition that visible patches within each local window of arbitrary size can be grouped with equal size, where masked self-attention is then performed within each group. Second, we further improve the grouping strategy via the Dynamic Programming algorithm to minimize the overall computation cost of the attention on the grouped patches. Third, as for the convolution layers, we convert them to the Sparse Convolution that works seamlessly with the sparse data, i. e. , the visible patches in MIM. As a result, MIM can now work on most, if not all, hierarchical ViTs in a green and efficient way. For example, we can train the hierarchical ViTs, e. g. , Swin Transformer and Twins Transformer, about 2. 7$\times$ faster and reduce the GPU memory usage by 70%, while still enjoying competitive performance on ImageNet classification and the superiority on downstream COCO object detection benchmarks.

PDF Details

NeurIPS Conference 2022 Conference Paper

Knowledge Distillation from A Stronger Teacher

  • Tao Huang
  • Shan You
  • Fei Wang
  • Chen Qian
  • Chang Xu

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https: //github. com/hunto/DIST_KD.

PDF Details

NeurIPS Conference 2022 Conference Paper

Weak-shot Semantic Segmentation via Dual Similarity Transfer

  • Junjie Chen
  • Li Niu
  • Siyuan Zhou
  • Jianlou Si
  • Chen Qian
  • Liqing Zhang

Semantic segmentation is a practical and active task, but severely suffers from the expensive cost of pixel-level labels when extending to more classes in wider applications. To this end, we focus on the problem named weak-shot semantic segmentation, where the novel classes are learnt from cheaper image-level labels with the support of base classes having off-the-shelf pixel-level labels. To tackle this problem, we propose a dual similarity transfer framework, which is built upon MaskFormer to disentangle the semantic segmentation task into single-label classification and binary segmentation for each proposal. Specifically, the binary segmentation sub-task allows proposal-pixel similarity transfer from base classes to novel classes, which enables the mask learning of novel classes. We also learn pixel-pixel similarity from base classes and distill such class-agnostic semantic similarity to the semantic masks of novel classes, which regularizes the segmentation model with pixel-level semantic relationship across images. In addition, we propose a complementary loss to facilitate the learning of novel classes. Comprehensive experiments on the challenging COCO-Stuff-10K and ADE20K datasets demonstrate the effectiveness of our method.

PDF Details

AAAI Conference 2021 Conference Paper

Conceptualized and Contextualized Gaussian Embedding

  • Chen Qian
  • Fuli Feng
  • Lijie Wen
  • Tat-Seng Chua

Word embedding can represent a word as a point vector or a Gaussian distribution in high-dimensional spaces. Gaussian distribution is innately more expressive than point vector owing to the ability to additionally capture semantic uncertainties of words, and thus can express asymmetric relations among words more naturally (e. g. , animal entails cat but not the reverse). However, previous Gaussian embedders neglect inner-word conceptual knowledge and lack tailored Gaussian contextualizer, leading to inferior performance on both intrinsic (context-agnostic) and extrinsic (context-sensitive) tasks. In this paper, we first propose a novel Gaussian embedder which explicitly accounts for innerword conceptual units (sememes) to represent word semantics more precisely; during learning, we propose Gaussian Distribution Attention over Gaussian representations to adaptively aggregate multiple sememe distributions into a word distribution, which guarantees the Gaussian linear combination property. Additionally, we propose a Gaussian contextualizer to utilize outer-word contexts in a sentence, producing contextualized Gaussian representations for contextsensitive tasks. Extensive experiments on intrinsic and extrinsic tasks demonstrate the effectiveness of the proposed approach, achieving state-of-the-art performance with near 5. 00% relative improvement.

PDF Details

AAAI Conference 2021 Conference Paper

Knowledge-aware Named Entity Recognition with Alleviating Heterogeneity

  • Binling Nie
  • Ruixue Ding
  • Pengjun Xie
  • Fei Huang
  • Chen Qian
  • Luo Si

Named Entity Recognition (NER) is a fundamental and important research topic for many downstream NLP tasks, aiming at detecting and classifying named entities (NEs) mentioned in unstructured text into pre-defined categories. Learning from labeled data only is far from enough when it comes to domain-specific or temporally-evolving entities (e. g. medical terminologies or restaurant names). Luckily, open-source Knowledge Bases (KBs) (e. g. Wikidata and Freebase) contain NEs that are manually labeled with predefined types in different domains, which is potentially beneficial to identify entity boundaries and recognize entity types more accurately. However, the type system of a domain-specific NER task is typically independent of that of current KBs and thus exhibits heterogeneity issue inevitably, which makes matching between the original NER and KB types (e. g. Person in NER potentially matches President in KBs) less likely, or introduces unintended noises without considering domainspecific knowledge (e. g. Band in NER should be mapped to Out of Entity Types in the restaurant-related task). To better incorporate and denoise the abundant knowledge in KBs, we propose a new KB-aware NER framework (KaNa), which utilizes type-heterogeneous knowledge to improve NER. Specifically, for an entity mention along with a set of candidate entities that are linked from KBs, KaNa first uses a type projection mechanism that maps the mention type and entity types into a shared space to homogenize the heterogeneous entity types. Then, based on projected types, a noise detector filters out certain less-confident candidate entities in an unsupervised manner. Finally, the filtered mention-entity pairs are injected into a NER model as a graph to predict answers. The experimental results demonstrate KaNa’s state-ofthe-art performance on five public benchmark datasets from different domain.

PDF Details

NeurIPS Conference 2021 Conference Paper

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

  • Mingkai Zheng
  • Shan You
  • Fei Wang
  • Chen Qian
  • Changshui Zhang
  • Xiaogang Wang
  • Chang Xu

Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most of methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduced a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. Moreover, to boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. Experimental results show that our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.

PDF Details

AAAI Conference 2020 Conference Paper

A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation

  • Yihua Cheng
  • Shiyao Huang
  • Fei Wang
  • Chen Qian
  • Feng Lu

Human gaze is essential for various appealing applications. Aiming at more accurate gaze estimation, a series of recent works propose to utilize face and eye images simultaneously. Nevertheless, face and eye images only serve as independent or parallel feature sources in those works, the intrinsic correlation between their features is overlooked. In this paper we make the following contributions: 1) We propose a coarseto-fine strategy which estimates a basic gaze direction from face image and refines it with corresponding residual predicted from eye images. 2) Guided by the proposed strategy, we design a framework which introduces a bi-gram model to bridge gaze residual and basic gaze direction, and an attention component to adaptively acquire suitable fine-grained feature. 3) Integrating the above innovations, we construct a coarse-to-fine adaptive network named CA-Net and achieve state-of-the-art performances on MPIIGaze and EyeDiap.

PDF Details

NeurIPS Conference 2020 Conference Paper

Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space

  • Shangchen Du
  • Shan You
  • Xiaojie Li
  • Jianlong Wu
  • Fei Wang
  • Chen Qian
  • Changshui Zhang

Distilling knowledge from an ensemble of teacher models is expected to have a more promising performance than that from a single one. Current methods mainly adopt a vanilla average rule, i. e. , to simply take the average of all teacher losses for training the student network. However, this approach treats teachers equally and ignores the diversity among them. When conflicts or competitions exist among teachers, which is common, the inner compromise might hurt the distillation performance. In this paper, we examine the diversity of teacher models in the gradient space and regard the ensemble knowledge distillation as a multi-objective optimization problem so that we can determine a better optimization direction for the training of student network. Besides, we also introduce a tolerance parameter to accommodate disagreement among teachers. In this way, our method can be seen as a dynamic weighting method for each teacher in the ensemble. Extensive experiments validate the effectiveness of our method for both logits-based and feature-based cases.

PDF Details

NeurIPS Conference 2020 Conference Paper

AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

  • Hao Zhu
  • Chaoyou Fu
  • Qianyi Wu
  • Wayne Wu
  • Chen Qian
  • Ran He

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.

PDF Details

NeurIPS Conference 2020 Conference Paper

ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

  • Yibo Yang
  • Hongyang Li
  • Shan You
  • Fei Wang
  • Chen Qian
  • Zhouchen Lin

Neural architecture search (NAS) aims to produce the optimal sparse solution from a high-dimensional space spanned by all candidate connections. Current gradient-based NAS methods commonly ignore the constraint of sparsity in the search phase, but project the optimized solution onto a sparse one by post-processing. As a result, the dense super-net for search is inefficient to train and has a gap with the projected architecture for evaluation. In this paper, we formulate neural architecture search as a sparse coding problem. We perform the differentiable search on a compressed lower-dimensional space that has the same validation loss as the original sparse solution space, and recover an architecture by solving the sparse coding problem. The differentiable search and architecture recovery are optimized in an alternate manner. By doing so, our network for search at each update satisfies the sparsity constraint and is efficient to train. In order to also eliminate the depth and width gap between the network in search and the target-net in evaluation, we further propose a method to search and evaluate in one stage under the target-net settings. When training finishes, architecture variables are absorbed into network weights. Thus we get the searched architecture and optimized parameters in a single run. In experiments, our two-stage method on CIFAR-10 requires only 0. 05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.

PDF Details

AAAI Conference 2020 Conference Paper

Reborn Filters: Pruning Convolutional Neural Networks with Limited Data

  • Yehui Tang
  • Shan You
  • Chang Xu
  • Jin Han
  • Chen Qian
  • Boxin Shi
  • Chao Xu
  • Changshui Zhang

Channel pruning is effective in compressing the pretrained CNNs for their deployment on low-end edge devices. Most existing methods independently prune some of the original channels and need the complete original dataset to fix the performance drop after pruning. However, due to commercial protection or data privacy, users may only have access to a tiny portion of training examples, which could be insufficient for the performance recovery. In this paper, for pruning with limited data, we propose to use all original filters to directly develop new compact filters, named reborn filters, so that all useful structure priors in the original filters can be well preserved into the pruned networks, alleviating the performance drop accordingly. During training, reborn filters can be easily implemented via 1 × 1 convolutional layers and then be fused in the inference stage for acceleration. Based on reborn filters, the proposed channel pruning algorithm shows its effectiveness and superiority on extensive experiments.

PDF Details

AAAI Conference 2020 Conference Paper

Solving Sequential Text Classification as Board-Game Playing

  • Chen Qian
  • Fuli Feng
  • Lijie Wen
  • Zhenpeng Chen
  • Li Lin
  • Yanan Zheng
  • Tat-Seng Chua

Sequential Text Classification (STC) aims to classify a sequence of text fragments (e. g. , words in a sentence or sentences in a document) into a sequence of labels. In addition to the intra-fragment text contents, considering the interfragment context dependencies is also important for STC. Previous sequence labeling approaches largely generate a sequence of labels in left-to-right reading order. However, the need for context information in making decisions varies across different fragments and is not strictly organized in a left-to-right order. Therefore, it is appealing to label the fragments that need less consideration of context information first before labeling the fragments that need more. In this paper, we propose a novel model that labels a sequence of fragments in jumping order. Specifically, we devise a dedicated boardgame to develop a correspondence between solving STC and board-game playing. By defining proper game rules and devising a game state evaluator in which context clues are injected, at each round, each player is effectively pushed to find the optimal move without position restrictions via considering the current game state, which corresponds to producing a label for an unlabeled fragment jumpily with the consideration of the contexts clues. The final game-end state is viewed as the optimal label sequence. Extensive results on three representative datasets show that the proposed approach outperforms the state-of-the-art methods with statistical significance.

PDF Details

AAAI Conference 2019 Conference Paper

Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation

  • Wei Feng
  • Wentao Liu
  • Tong Li
  • Jing Peng
  • Chen Qian
  • Xiaolin Hu

Human-object interactions (HOI) recognition and pose estimation are two closely related tasks. Human pose is an essential cue for recognizing actions and localizing the interacted objects. Meanwhile, human action and their interacted objects’ localizations provide guidance for pose estimation. In this paper, we propose a turbo learning framework to perform HOI recognition and pose estimation simultaneously. First, two modules are designed to enforce message passing between the tasks, i. e. pose aware HOI recognition module and HOI guided pose estimation module. Then, these two modules form a closed loop to utilize the complementary information iteratively, which can be trained in an end-to-end manner. The proposed method achieves the state-of-the-art performance on two public benchmarks including Verbs in COCO (V-COCO) and HICO-DET datasets.

PDF Details

AAAI Conference 2018 Conference Paper

A Cascaded Inception of Inception Network With Attention Modulated Feature Fusion for Human Pose Estimation

  • Wentao Liu
  • Jie Chen
  • Cheng Li
  • Chen Qian
  • Xiao Chu
  • Xiaolin Hu

Accurate keypoint localization of human pose needs diversified features: the high level for contextual dependencies and the low level for detailed refinement of joints. However, the importance of the two factors varies from case to case, but how to efficiently use the features is still an open problem. Existing methods have limitations in preserving low level features, adaptively adjusting the importance of different levels of features, and modeling the human perception process. This paper presents three novel techniques step by step to efficiently utilize different levels of features for human pose estimation. Firstly, an inception of inception (IOI) block is designed to emphasize the low level features. Secondly, an attention mechanism is proposed to adjust the importance of individual levels according to the context. Thirdly, a cascaded network is proposed to sequentially localize the joints to enforce message passing from joints of stand-alone parts like head and torso to remote joints like wrist or ankle. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance on both MPII and LSP benchmarks.

PDF Details

IJCAI Conference 2018 Conference Paper

DRPose3D: Depth Ranking in 3D Human Pose Estimation

  • Min Wang
  • Xipeng Chen
  • Wentao Liu
  • Chen Qian
  • Liang Lin
  • Lizhuang Ma

In this paper, we propose a two-stage depth ranking based method (DRPose3D) to tackle the problem of 3D human pose estimation. Instead of accurate 3D positions, the depth ranking can be identified by human intuitively and learned using the deep neural network more easily by solving classification problems. Moreover, depth ranking contains rich 3D information. It prevents the 2D-to-3D pose regression in two-stage methods from being ill-posed. In our method, firstly, we design a Pairwise Ranking Convolutional Neural Network (PRCNN) to extract depth rankings of human joints from images. Secondly, a coarse-to-fine 3D Pose Network(DPNet) is proposed to estimate 3D poses from both depth rankings and 2D human joint locations. Additionally, to improve the generality of our model, we introduce a statistical method to augment depth rankings. Our approach outperforms the state-of-the-art methods in the Human3. 6M benchmark for all three testing protocols, indicating that depth ranking is an essential geometric feature which can be learned to improve the 3D pose estimation.

PDF Details
v2026.09.13