Author name cluster

Dan Xu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

21 papers

2 author rows

AAAI Conference 2026 Conference Paper

Emotion-Conditioned Motion Sub-spaces with Flow Matching for Real-Time Audio-Driven Talking Heads

Haoyu Wang
Xiaozhe Xin
Xiaoyu Qin
Meiguang Jin
Junfeng Ma
Dan Xu
Jia Jia

Recent advances in audio-driven talking-head synthesis have brought lip-sync precision close to human perception, yet emotional fidelity and real-time inference remain open challenges. Existing pipelines typically disentangle lip articulation, facial expression, and head pose in latent space; this rigid factorization ignores the intrinsic coupling between articulation and affect — e.g., downward lip corners when sad—thus limiting expressiveness. We cast speech-conditioned facial motion as a sample from an emotion-conditioned distribution in a motion latent space. Concretely, we (i) learn a motion dictionary of orthogonal bases with an autoencoder via self-supervision, (ii) construct emotion-conditioned sub-spaces within the latent space, and (iii) design a layer-progressive cross-attention fusion module that modulates a flow-matching sampler with both audio and emotion signals. Only ten reverse ODE steps are required to generate a motion-latent trajectory, enabling real-time end-to-end latency. Extensive experiments on MEAD and RAVDESS show that our method outperforms recent GAN- and diffusion-based baselines in emotion accuracy while running at around 75 FPS on a single desktop GPU. The proposed framework delivers the first emotionally expressive Audio2Face system that simultaneously achieves lip-sync accuracy, affective realism, and real-time performance.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Empowering Sparse-Input Neural Radiance Fields with Dual-Level Semantic Guidance from Dense Novel Views

Yingji Zhong
Kaichen Zhou
Zhihao Li
Lanqing Hong
Zhenguo Li
Dan Xu

Neural Radiance Fields (NeRF) have shown remarkable capabilities for photorealistic novel view synthesis. One major deficiency of NeRF is that dense inputs are typically required, and the rendering quality will drop drastically given sparse inputs. In this paper, we highlight the effectiveness of rendered semantics from dense novel views, and show that rendered semantics can be treated as a more robust form of augmented data than rendered RGB. Our method enhances NeRF’s performance by incorporating guidance derived from the rendered semantics. The rendered semantic guidance encompasses two levels: the supervision level and the feature level. The supervision-level guidance incorporates a bi-directional verification module that decides the validity of each rendered semantic label, while the feature-level guidance integrates a learnable codebook that encodes semantic-aware information, which is queried by each point via the attention mechanism to obtain semanticrelevant predictions. The overall semantic guidance is embedded into a self-improved pipeline.We also introduce a more challenging sparse-input indoor benchmark, where the number of inputs is limited to as few as 6. Experiments demonstrate the effectiveness of our method and it exhibits superior performance compared to existing approaches.

PDF Details DOI

EAAI Journal 2025 Journal Article

An invulnerable leader–follower collision-free unmanned aerial vehicle flocking system with attention-based Multi-Agent Reinforcement Learning

Yunxiao Guo
Dan Xu
Chang Wang
Jinxi Li
Han Long

Deep reinforcement learning has been proved useful for the flocking control of Unmanned Aerial Vehicle (UAV) swarm with the leader–followers topology. However, it remains unclear how to fully utilize the spatial information among the followers to alleviate the problems of sparse reward and policy convergence. In this article, we propose a novel multi-agent reinforcement learning-based fixed-wing UAV flocking approach named Attention Based Cucker–Smale (ABCS) Flocking to learn collision-free leader–follower flocking by utilizing the information among followers with the attention mechanism. Specifically, we design an explainable Cucker–Smale criterion-based flocking reward named ABCS reward to associate the followers with high efficiency in flocking. Then, a leader-guide attention mechanism is proposed by transferring the difference between leader and follower as the weights to support the follower selectively utilizing the followers’ information. As a result, we prove that an optimal state can be achieved so that each follower can keep an optimal distance from the other followers when the ABCS reward is maximized. In addition, we prove the ABCS reward is bounded, which can be used to indicate learning convergence. To improve the invulnerability, we proposes a leader selection method based on ABCS flocking, which can effectively select a new leader when the old leader is destroyed. Finally, we demonstrate the effectiveness of ABCS Flocking over the Multi-Agent Deep Deterministic Policy Gradient approach using various reward functions with various numbers of followers in the obstacles and leader-destroyed scenarios. The code is published on github https: //github. com/YunxiaoGuo/ABCS-Flocking.

Details DOI

IJCAI Conference 2025 Conference Paper

Human-Centric Foundation Models: Perception, Generation and Agentic Modeling

Shixiang Tang
Yizhou Wang
Lu Chen
Yuan Wang
Sida Peng
Dan Xu
Wanli Ouyang

Human understanding and generation are critical for modeling digital humans and humanoid embodiments. Recently, Human-centric Foundation Models (HcFMs)—inspired by the success of generalist models such as large language and vision models—have emerged to unify diverse human-centric tasks into a single framework, surpassing traditional task-specific approaches. In this survey, we present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups: (1) Human-centric Perception Foundation Models that capture fine-grained features for multi-modal 2D and 3D understanding; (2) Human-centric AIGC Foundation Models that generate high-fidelity, diverse human-related content; (3) Unified Perception and Generation Models that integrate these capabilities to enhance both human understanding and synthesis; and (4) Human-centric Agentic Foundation Models that extend beyond perception and generation to learn human-like intelligence and interactive behaviors for humanoid embodied tasks. We review state-of-the-art techniques, discuss emerging challenges and future research directions. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling. Website is https: //github. com/HumanCentricModels/Awesome-Human-Centric-Foundation-Models/

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

HyRF: Hybrid Radiance Fields for Memory-efficient and High-quality Novel View Synthesis

Zipeng Wang
Dan Xu

Recently, 3D Gaussian Splatting (3DGS) has emerged as a powerful alternative to NeRF-based approaches, enabling real-time, high-quality novel view synthesis through explicit, optimizable 3D Gaussians. However, 3DGS suffers from significant memory overhead due to its reliance on per-Gaussian parameters to model view-dependent effects and anisotropic shapes. While recent works propose compressing 3DGS with neural fields, these methods struggle to capture high-frequency spatial variations in Gaussian properties, leading to degraded reconstruction of fine details. We present Hybrid Radiance Fields (HyRF), a novel scene representation that combines the strengths of explicit Gaussians and neural fields. HyRF decomposes the scene into (1) a compact set of explicit Gaussians storing only critical high-frequency parameters and (2) grid-based neural fields that predict remaining properties. To enhance representational capacity, we introduce a decoupled neural field architecture, separately modeling geometry (scale, opacity, rotation) and view-dependent color. Additionally, we propose a hybrid rendering scheme that composites Gaussian splatting with a neural field-predicted background, addressing limitations in distant scene representation. Experiments demonstrate that HyRF achieves state-of-the-art rendering quality while reducing model size by over 20× compared to 3DGS and maintaining real-time performance.

PDF Details

AAAI Conference 2025 Conference Paper

Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective

You Zhang
Jin Wang
Liang-Chih Yu
Dan Xu
Xuejie Zhang

Current neural networks often employ multi-domain-learning or attribute-injecting mechanisms to incorporate non-independent and identically distributed (non-IID) information for text understanding tasks by capturing individual characteristics and the relationships among samples. However, the extent of the impact of non-IID information and how these methods affect pre-trained language models (PLMs) remains unclear. This study revisits the assumption that non-IID information enhances PLMs to achieve performance improvements from a Bayesian perspective, which unearths and integrates non-IID and IID features. Furthermore, we proposed a multi-attribute multi-grained framework for PLM adaptations (M2A), which combines multi-attribute and multi-grained views to mitigate uncertainty in a lightweight manner. We evaluate M2A through prevalent text-understanding datasets and demonstrate its superior performance, mainly when data are implicitly non-IID, and PLMs scale larger.

PDF Details DOI

EAAI Journal 2025 Journal Article

TSSA-Net: Transposed Sparse Self-Attention-based network for image super-resolution

Guanhao Chen
Dan Xu
Kangjian He
Hongzhen Shi
Hao Zhang

Image super-resolution (SR) aims to reconstruct high-resolution images from low-resolution ones. Transformer-based methods have recently demonstrated remarkable results, but the conventional dense self-attention mechanism fails to capture local relationships between patches, leading to suboptimal performance. Moreover, the recovery of high-frequency information, crucial for edge reconstruction, is often insufficient. To address these challenges, we propose the Transposed Sparse Self-Attention (TSSA) mechanism, which improves local feature attention by restructuring the self-attention computation, without using convolutions. Additionally, we introduce a Segmented Convolutional Feed-Forward Network (SCFFN) to enhance high-frequency detail recovery and local feature acquisition, while maintaining a low parameter count. We combine TSSA, SCFFN, and a channel attention mechanism to develop TSSA-Net, an innovative network for image super-resolution. Comprehensive evaluations on classical, lightweight, and real-world SR tasks show that TSSA-Net outperforms recent methods on the Set14, Urban100, and Manga109 datasets, with improvements of 0. 01–0. 04 decibel (dB), 0. 11–0. 12 dB, and 0. 01–0. 08 dB, respectively. TSSA-Net achieves notable results in both metric-based and visual-based evaluations. The code is available at https: //github. com/VMC-Lab-Chen/TSSA-Net.

Details DOI

AAAI Conference 2025 Conference Paper

Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation

Kuanghong Liu
Jin Wang
Kangjian He
Dan Xu
Xuejie Zhang

Conventional multi-source domain few-shot adaptation (MFDA) faces the challenge of further reducing the load on edge-side devices in low-resource scenarios. Considering the native language-supervised advantage of CLIP and the plug-and-play nature of prompt to transfer CLIP efficiently, this paper introduces an uploadable multi-source few-shot domain adaptation (UMFDA) schema. It belongs to a decentralized edge collaborative learning in the edge-side models that must maintain a low computational load. And only a limited amount of annotations in source domain data is provided, with most of the data being unannotated. Further, this paper proposes a vision-aware multimodal prompt tuning framework (VAMP) under the decentralized schema, where the vision-aware prompt guides the text domain-specific prompt to maintain semantic discriminability and perceive the domain information. The cross-modal semantic and domain distribution alignment losses optimize each edge-side model, while text classifier consistency and semantic diversity losses promote collaborative learning among edge-side models. Extensive experiments were conducted on OfficeHome and DomainNet datasets to demonstrate the effectiveness of the proposed VAMP in the UMFDA, which outperformed the previous prompt tuning methods.

PDF Details DOI

IROS Conference 2024 Conference Paper

A Novel Variable Step-size Path Planning Framework with Step-Consistent Markov Decision Process For Large-Scale UAV Swarm

Dan Xu
Yunxiao Guo
Han Long
Chang Wang

In recent years, Deep Reinforcement Learning (DRL) has been a key approach to solving Unmanned Aerial Vehicle (UAV) swarm path planning problems. However, traditional DRL methods often face challenges in the initial learning stage and struggle to learn from variable step-size tasks. This paper introduces a novel training framework for large-scale UAV swarm variable step-size path planning: Rapidly-exploring Variable Step-size Deep Reinforcement Learning (RVSDRL). This framework involves common training on the ground local server and decentralized training on distributed UAVs. In the common training stage, we generate rapidly-exploring random graph samples to accelerate the common agent explore environment. In the decentralized training stages, we utilize the priority replay mechanism to improve efficiency. To enhance convergence stability, we restrict the returns of the equivalent paths and propose the Step-size Consistent Markov Decision Process (SCMDP) path planning model. Our method is compared with traditional methods, and the experiments demonstrate its superior performance in complex obstacle environments.

Details

EAAI Journal 2024 Journal Article

ASFusion: Adaptive visual enhancement and structural patch decomposition for infrared and visible image fusion

Yiqiao Zhou
Kangjian He
Dan Xu
Dapeng Tao
Xu Lin
Chengzhou Li

Multimodal data fusion plays an increasingly important role in the field of artificial intelligence. The objective of Infrared and Visible Image Fusion (IVF) is to integrate information from different types of images to enhance the performance of target detection tasks. Meanwhile, object detection technology constitutes a crucial foundation in the field of autonomous driving. However, visible images captured under low illumination often lack important details, resulting in suboptimal fusion results，which in turn affects the accuracy of target detection tasks. We proposed an infrared and visible image fusion method based on adaptive visual enhancement and structural patch decomposition (ASFusion) to address the above issues. First, we design an efficient algorithm based on the camera response model to enhance different exposure matrices, allowing for adaptive enhancement of visible images. Second, we decompose the source infrared and the enhanced visible image into three components: mean intensity, signal structure, and signal intensity using structural patch decomposition (SPD), and then design a new degree of membership curve function to estimate the weight of the average intensity component accurately. The estimation process reduces artifacts and preserves the significance of infrared targets. Third, to achieve a higher contrast in the fusion result, we introduced sharpening operations to enhance the detail layer of both the infrared and visible images. Finally, the fused image is obtained by merging the base and detail layers. Through qualitative and quantitative experimental evaluations, the proposed method outperforms twelve state-of-the-art image fusion methods. Additionally, object detection experiments have demonstrated that our ASFusion exhibits tremendous potential in better serving advanced computer vision tasks. Our code is publicly available at https: //github. com/ZhouVMC/ASFusion.

Details DOI

EAAI Journal 2024 Journal Article

FCLFusion: A frequency-aware and collaborative learning for infrared and visible image fusion

Chengchao Wang
Yuanyuan Pu
Zhengpeng Zhao
Rencan Nie
Jinde Cao
Dan Xu

Infrared and visible image fusion (IVIF) aims to integrate the advantages of different modal images. Most existing deep learning-based methods often focus on a single IVIF task and ignore the effect of frequency information on the fusion results, which do not fully preserve saliency structures and important texture details. The core idea of this paper is based on the following observation: (1) image content can be characterized by different frequency domain components, low frequency represents base information, such as saliency structure, while high frequency contains texture detail information. (2) multi-tasks learning generally achieve better performance than single-task. Based on these observations, we propose a fusion model called Frequency-aware and Collaborative Learning (FCLFusion) for infrared and visible images. This model takes image fusion as the main task and introduces image reconstruction as an auxiliary task to collaboratively optimize the network, thereby improving the fusion quality. Specifically, we transform spatial domain features to the frequency domain and develop a frequency feature fusion module for guiding the primary network to generate the fused image. The sub-network generates the reconstructed images. Also, we preserve the saliency and detail features via frequency skip connections. Moreover, we propose a hybrid loss function that consists of two terms: frequency loss and self-supervised reconstruction loss. The former aims to prevent information loss in the frequency domain, while the latter improves the extraction of vital information. Extensive experiments verified on three public datasets demonstrate that our FCLFusion outperforms ten state-of-the-art fusion models.

Details DOI

EAAI Journal 2024 Journal Article

GCCF: A lightweight and scalable network for underwater image enhancement

Chufan Liu
Xin Shu
Dan Xu
Jinlong Shi

Underwater images suffer from color deviation and low contrast arising from light attenuation and scatter, making them act as a drag on exploring and exploiting ocean resources. Existed underwater image enhancement algorithms are either inflexible due to scene-dependent design or redundant for ignoring prior knowledge. To address the above problems, we propose a lightweight and scalable deep learning-based network that consists of a Grouped Color Compensation encoder and channel Fusion decoder, abbreviated GCCF. Concretely, the least absorbed green channel contains abundant information compared to the red and blue channels. Taking this into account, the encoder extracts information independently from three grouped features according to the color. Simultaneously, the encoder flexibly compensates for the red and blue channels from the green channel with a proposed learnable module. Then, the decoder gradually fuses the grouped features into a whole part and produces competitive outputs from the integrated features. Extensive experiments on real-world underwater images demonstrate that the proposed GCCF achieves outstanding results in subjective visual comparisons and objective metrics.

Details DOI

EAAI Journal 2024 Journal Article

Multivariable correlation feature network construction and health condition assessment for unlabeled single-sample data

Dan Xu
Xiaoqi Xiao
Jianguo Zhang

The construction of effective health indicators is crucial for assessing system degradation, enabling anomaly detection and health condition assessment, which contribute to reducing costs, improving productivity, and enhancing system availability. However, there are notable challenges in the health condition assessment for unlabeled single-sample data. In this paper, a multivariable correlation feature network—the Nested Autoencoder (NAE) network, is proposed, which incorporates inter-variable correlations to constrain the construction of latent space, thereby enhancing the accuracy of health condition assessment. Moreover, the autoencoder parameters calculated with each discrete interval data are leveraged to construct latent feature graphs, enabling component anomaly detection. Furthermore, a comprehensive indicator is introduced to describe the topological changes of the graphs, facilitating the assessment of the health conditions of the system. Finally, the effectiveness of our method is validated on the N-CMAPSS dataset as well as a real satellite dataset.

Details DOI

AAAI Conference 2024 Conference Paper

Personalized LoRA for Human-Centered Text Understanding

You Zhang
Jin Wang
Liang-Chih Yu
Dan Xu
Xuejie Zhang

Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.

PDF Details DOI

JBHI Journal 2024 Journal Article

RegFSC-Net: Medical Image Registration via Fourier Transform With Spatial Reorganization and Channel Refinement Network

Chenou Liu
Kangjian He
Dan Xu
Hongzhen Shi
Hao Zhang
Kunyuan Zhao

Medical image registration is crucial in medical image analysis applications. Recently, U-Net-style networks have been commonly used for unsupervised image registration, predicting dense displacement fields in full-resolution space. However, this process is resource-intensive and time-consuming for high-resolution volumetric image data. To address this challenge, this paper proposes a novel model named RegFSC-Net, which utilizes Fourier transform with spatial reorganization (SR) and channel refinement (CR) network for registration. We embed efficient feature extraction modules SR and CR modules into the encoder, and adopt a parameter-free model to drive the decoder to improve the U-shaped network. Precisely, RegFSC-Net does not directly predict the full-resolution displacement field in space but learns the low-dimensional representation of the displacement field in the bandlimited Fourier domain, which is beneficial in reducing network parameters, memory usage, and computational costs. Experimental results show that RegFSC-Net outperforms various state-of-the-art methods. Specifically, in comparison to the widely recognized Transformer-based method TransMorph, RegFSC-Net utilizes only around 8. 2% of its parameters, resulting in a 1. 95% higher Dice score and significantly faster inference speeds of 126. 67% and 419. 99% on GPU and CPU, respectively. Furthermore, we also designed three variants of RegFSC-Net and demonstrated their potential applications in computer-aided diagnosis.

Details DOI

EAAI Journal 2024 Journal Article

Transformer framework for depth-assisted UDA semantic segmentation

Yunna Song
Jinlong Shi
Danping Zou
Caisheng Liu
Suqin Bai
Xin Shu
Qiang Qian
Dan Xu

Unsupervised domain adaptation (UDA) plays a crucial role in transferring models trained on synthetic datasets to real-world datasets. In semantic segmentation, UDA can alleviate the requirement of a large number of dense semantic annotations. Some UDA semantic segmentation approaches have already leveraged depth information to enhance semantic features for improved segmentation accuracy. Building on this, we introduce a UDA multitask Transformer framework called Multi-former. Multi-former contains a semantic-segmentation and a depth-estimation network. Depth-estimation network extracts more informative depth features to estimate depth and assist in semantic segmentation. In addition, considering the issue of imbalanced class pixel distributions in the source domain, we present a rare class mix strategy (RCM) to balance domain adaptability for all classes. To further enhance the UDA semantic segmentation performance, we design a mixed label loss weight strategy (MLW), which employs different types of weights to comprehensively utilize the features of pseudo-label. Experimental results demonstrate the effectiveness of the proposed approach, which achieves the best mean intersection over union (mIoU) of 56. 1% and 76. 3% on the two UDA benchmark tasks of synthetic datasets to real-world datasets, respectively. The code and models are available at https: //github. com/fz-ss/Multi-former.

Details DOI

NeurIPS Conference 2023 Conference Paper

CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection

Yang Cao
Zeng Yihan
Hang Xu
Dan Xu

Open-vocabulary 3D Object Detection (OV-3DDet) aims to detect objects from an arbitrary list of categories within a 3D scene, which remains seldom explored in the literature. There are primarily two fundamental problems in OV-3DDet, i. e. , localizing and classifying novel objects. This paper aims at addressing the two problems simultaneously via a unified framework, under the condition of limited base categories. To localize novel 3D objects, we propose an effective 3D Novel Object Discovery strategy, which utilizes both the 3D box geometry priors and 2D semantic open-vocabulary priors to generate pseudo box labels of the novel objects. To classify novel object boxes, we further develop a cross-modal alignment module based on discovered novel boxes, to align feature spaces between 3D pointcloud and image/text modalities. Specifically, the alignment process contains a class-agnostic and a class-discriminative alignment, incorporating not only the base objects with annotations but also the increasingly discovered novel objects, resulting in an iteratively enhanced alignment. The novel box discovery and crossmodal alignment are jointly learned to collaboratively benefit each other. Thenovel object discovery can directly impact the cross-modal alignment, while a better feature alignment can, in turn, boost the localization capability, leading to a unified OV-3DDet framework, named CoDA, for simultaneous novel object localization and classification. Extensive experiments on two challenging datasets ( i. e. , SUN-RGBD and ScanNet) demonstrate the effectiveness of our method and also show a significant mAP improvement upon the best-performing alternative method by 80%. Codes and pre-trained models are released on the project page.

PDF Details

AAAI Conference 2023 Conference Paper

Contrastive Multi-Task Dense Prediction

Siwei Yang
Hanrong Ye
Dan Xu

This paper targets the problem of multi-task dense prediction which aims to achieve simultaneous learning and inference on a bunch of multiple dense prediction tasks in a single framework. A core objective in design is how to effectively model cross-task interactions to achieve a comprehensive improvement on different tasks based on their inherent complementarity and consistency. Existing works typically design extra expensive distillation modules to perform explicit interaction computations among different task-specific features in both training and inference, bringing difficulty in adaptation for different task sets, and reducing efficiency due to clearly increased size of multi-task models. In contrast, we introduce feature-wise contrastive consistency into modeling the cross-task interactions for multi-task dense prediction. We propose a novel multi-task contrastive regularization method based on the consistency to effectively boost the representation learning of the different sub-tasks, which can also be easily generalized to different multi-task dense prediction frameworks, and costs no additional computation in the inference. Extensive experiments on two challenging datasets (i.e. NYUD-v2 and Pascal-Context) clearly demonstrate the superiority of the proposed multi-task contrastive learning approach for dense predictions, establishing new state-of-the-art performances.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

Lewei Yao
Jianhua Han
Youpeng Wen
Xiaodan Liang
Dan Xu
Wei Zhang
Zhenguo Li
Chunjing Xu

Open-world object detection, as a more general and challenging goal, aims to recognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem as a grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately to better utilize heterogeneous datasets (i. e. , detection, grounding, and image-text pairs) for training. We further design a concept dictionary (with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction of the word-region alignment loss, and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, e. g. , on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9. 9% mAP and obtains a 13. 5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.

PDF Details

YNIMG Journal 2021 Journal Article

Development of functional connectivity within and among the resting-state networks in anesthetized rhesus monkeys

Bo Rao
Dan Xu
Chaoyang Zhao
Shouchao Wang
Xuan Li
Wenbo Sun
Yadong Gang
Jian Fang

OBJECTIVE: The age-related changes in the resting-state networks (RSNs) exhibited temporally specific patterns in humans, and humans and rhesus monkeys have similar RSNs. We hypothesized that the RSNs in rhesus monkeys experienced similar developmental patterns as humans. METHODS: We acquired resting-state fMRI data from 62 rhesus monkeys, which were divided into childhood, adolescence, and early adulthood groups. Group independent component analysis (ICA) was used to identify monkey RSNs. We detected the between-group differences in the RSNs and static, dynamic, and effective functional network connections (FNCs) using one-way variance analysis (ANOVA) and post-hoc analysis. RESULTS: Eight rhesus RSNs were identified, including cerebellum (CN), left and right lateral visual (LVN and RVN), posterior default mode (pDMN), visuospatial (VSN), frontal (FN), salience (SN), and sensorimotor networks (SMN). In internal connections, the CN, SN, FN, and SMN mainly matured in early adulthood. The static FNCs associated with FN, SN, pDMN primarily experienced fast descending slow ascending type (U-shaped) developmental patterns for maturation, and the dynamic FNCs related to pDMN (RVN, CN, and SMN) and SMN (CN) were mature in early adulthood. The effective FNC results showed that the pDMN and VSN (stimulated), SN (inhibited), and FN (first inhibited then stimulated) chiefly matured in early adulthood. CONCLUSION: We identified eight monkey RSNs, which exhibited similar development patterns as humans. All the RSNs and FNCs in monkeys were not widely changed but fine-tuned. Our study clarified that the progressive synchronization, exploration, and regulation of cognitive RSNs within the pDMN, FN, SN, and VSN denoted potential maturation of the RSNs throughout development. We confirmed the development patterns of RSNs and FNCs would support the use of monkeys as a best animal model for human brain function.

Details DOI

NeurIPS Conference 2017 Conference Paper

Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction

Dan Xu
Wanli Ouyang
Xavier Alameda-Pineda
Elisa Ricci
Xiaogang Wang
Nicu Sebe

Recent works have shown that exploiting multi-scale representations deeply learned via convolutional neural networks (CNN) is of tremendous importance for accurate contour detection. This paper presents a novel approach for predicting contours which advances the state of the art in two fundamental aspects, i. e. multi-scale feature generation and fusion. Different from previous works directly considering multi-scale feature maps obtained from the inner layers of a primary CNN architecture, we introduce a hierarchical deep model which produces more rich and complementary representations. Furthermore, to refine and robustly fuse the representations learned at different scales, the novel Attention-Gated Conditional Random Fields (AG-CRFs) are proposed. The experiments ran on two publicly available datasets (BSDS500 and NYUDv2) demonstrate the effectiveness of the latent AG-CRF model and of the overall hierarchical framework.

PDF Details