Arrow Research search

Author name cluster

Yue Gao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

61 papers
2 author rows

Possible papers

61

JBHI Journal 2026 Journal Article

An Experience-driven Interpretable Multi-task Model for Segmentation and Classification of Small Cell Lung Cancer and Non-small Cell Lung Cancer from CT Images

  • Zhaoshuo Diao
  • Manyu Cui
  • Taichang Xu
  • Ye Yuan
  • Guoyu Tong
  • Yue Gao

Lung cancer is the leading cause of cancer-related mortality, with small cell lung cancer and non-small cell lung cancer being the primary subtypes that exhibit distinct treatment approaches and prognostic outcomes. Accurate identification of these lung cancer classes holds significant importance in clinical practice. This study introduces an experience-driven interpretable multi-task network to concurrently perform segmentation and classification of small cell and non-small cell lung cancer. The core architecture of this multi-task model is based on StarNet, featuring a shared feature extraction branch and task-specific decoding branches for tumor segmentation and classification. Leveraging clinical knowledge of small cell lung cancer characteristics, such as indistinct edges, tissue invasion, and limited large cavity areas, two auxiliary branches are proposed: edge uncertainty estimation and tumor core area reconstruction. The values from edge uncertainty estimation and reconstruction integrity estimation are utilized in the classification branch to facilitate small cell lung cancer classification. Furthermore, for enhanced interpretability, bottleneck layer features are extracted for comparative learning, and a three-level contrastive loss is proposed to improve the differentiation of disease features. Lastly, an interpretable strategy based on trained feature query matching is presented, providing radiologists with clinical insights and reference images while the model outputs recognition predictions. Experimental results on the public dataset demonstrate that the proposed multi-task model not only outperforms single-task models but also offers a certain level of interpretability, thus enhancing radiologists' clinical decision-making processes.

Details DOI

AAAI Conference 2026 Conference Paper

Cog-RAG: Cognitive-Inspired Dual-Hypergraph with Theme Alignment Retrieval-Augmented Generation

  • Hao Hu
  • Yifan Feng
  • Ruoxue Li
  • Rundong Xue
  • Xingliang Hou
  • Zhiqiang Tian
  • Yue Gao
  • Shaoyi Du

Retrieval-Augmented Generation (RAG) enhances the response quality and domain-specific performance of large language models (LLMs) by incorporating external knowledge to combat hallucinations. In recent research, graph structures have been integrated into RAG to enhance the capture of semantic relations between entities. However, it primarily focuses on low-order pairwise entity relations, limiting the high-order associations among multiple entities. Hypergraph-enhanced approaches address this limitation by modeling multi-entity interactions via hyperedges, but they are typically constrained to inter-chunk entity-level representations, overlooking the global thematic organization and alignment across chunks. Drawing inspiration from the top-down cognitive process of human reasoning, we propose a theme-aligned dual-hypergraph RAG framework (Cog-RAG) that uses a theme hypergraph to capture inter-chunk thematic structure and an entity hypergraph to model high-order semantic relations. Furthermore, we design a cognitive-inspired two-stage retrieval strategy that first activates query-relevant thematic content from the theme hypergraph, and then guides fine-grained recall and diffusion in the entity hypergraph, achieving semantic alignment and consistent generation from global themes to local details. Our extensive experiments demonstrate that Cog-RAG significantly outperforms existing state-of-the-art baseline approaches.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Coordinated Humanoid Robot Locomotion with Symmetry Equivariant Reinforcement Learning Policy

  • Buqing Nie
  • Yang Zhang
  • Rongjun Jin
  • Zhanxiang Cao
  • Huangxuan Lin
  • Xiaokang Yang
  • Yue Gao

The human nervous system exhibits bilateral symmetry, enabling coordinated and balanced movements. However, existing Deep Reinforcement Learning (DRL) methods for humanoid robots neglect morphological symmetry of the robot, leading to uncoordinated and suboptimal behaviors. Inspired by human motor control, we propose Symmetry Equivariant Policy (SE-Policy), a new DRL framework that embeds strict symmetry equivariance in the actor and symmetry invariance in the critic without additional hyperparameters. SE-Policy enforces consistent behaviors across symmetric observations, producing temporally and spatially coordinated motions with higher task performance. Extensive experiments on velocity tracking tasks, conducted in both simulation and real-world deployment with the Unitree G1 humanoid robot, demonstrate that SE-Policy improves tracking accuracy by up to 40% compared to state-of-the-art baselines, while achieving superior spatial-temporal coordination. These results demonstrate the effectiveness of SE-Policy and its broad applicability to humanoid robots.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Keep On Going: Learning Robust Humanoid Motion Skills via Selective Adversarial Training

  • Yang Zhang
  • Zhanxiang Cao
  • Buqing Nie
  • Haoyang Li
  • Zhong Jiangwei
  • Qiao Sun
  • Xiaoyi Hu
  • Xiaokang Yang

Humanoid robots are expected to operate reliably over long horizons while executing versatile whole-body skills. Yet Reinforcement Learning (RL) motion policies typically lose stability under prolonged operation, sensor/actuator noise, and real world disturbances. In this work, we propose a Selective Adversarial Attack for Robust Training (SA2RT) to enhance the robustness of motion skills. The adversary is learned to identify and sparsely perturb the most vulnerable states and actions under an attack-budget constraint, thereby exposing true weakness without inducing conservative overfitting. The resulting non-zero sum, alternating optimization continually strengthens the motion policy against the strongest discovered attacks. We validate our approach on the Unitree G1 humanoid robot across perceptive locomotion and whole-body control tasks. Experimental results show that adversarially trained policies improve the terrain traversal success rate by 40%, reduce the trajectory tracking error by 32%, and maintain long horizon mobility and tracking performance. Together, these results demonstrate that selective adversarial attacks are an effective driver for learning robust, long horizon humanoid motion skills.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Role Hypergraph Contrastive Learning for Multivariate Time-Series Analysis

  • Rundong Xue
  • Hao Hu
  • Zhitao Zeng
  • Xiangmin Han
  • Zhiqiang Tian
  • Shaoyi Du
  • Yue Gao

Multivariate Time-Series (MTS) analysis is crucial across various domains. Considering the spatial and temporal consistency of MTS, existing methods leverage graph structures with temporal augmentation and contrastive learning to achieve robust learning of spatial dependencies and temporal patterns. Given the inherent high-order correlations in MTS, hypergraphs present a promising approach. However, two key challenges limit their further development: 1) Feature-based perspectives capture limited spatial information, while structural perspectives encode richer spatial consistency and evolution dependency; 2) Various semantic patterns (e.g., synergy, inhibition) entangle in sensor correlations, leading to semantic ambiguity. The underlying reason is that conventional hypergraph structures cannot distinguish specific semantic roles within or across hyperedges. Thus, we propose Role Hypergraph Contrastive Learning for MTS analysis. Specifically, we introduce the concept of role to generalize hypergraphs to Role Hypergraphs, enabling precise modeling of sensor correlations by assigning each vertex-hyperedge pair with a semantic role. Building on this structure, we design a role hypergraph contrastive learning paradigm to comprehensively capture the spatial and temporal dependencies: From a structural perspective, role hypergraph structural contrasting captures spatial short-term consistency and long-term evolution; from a feature perspective, alignment of complementary role information ensures sensor-level temporal consistency. Experiments on classification and forecasting tasks demonstrate the effectiveness and interpretability of our method.

PDF Details DOI

IROS Conference 2025 Conference Paper

Generalized Locomotion in Out-of-distribution Conditions with Robust Transformer

  • Lingxiao Guo
  • Yue Gao

To succeed in the real world, robots must deal with situations that differ from those seen during training. Those out-of-distribution situations for legged robot mainly include challenging dynamic gaps and perceptual gaps. Here we study the problem of robust locomotion in such novel situations. While previous methods usually rely on designing elaborate training and adaptation techniques, we approach the problem from a network model perspective. Our approach, RObust Locomotion Transformer(ROLT), a variation of transformer, could achieve robustness in a variety of unseen conditions. ROLT introduces two key designs: body tokenization and consistent dropout. Body tokenization supports knowledge share across different limbs, which boosts generalization ability of the network. Meanwhile, a novel dropout strategy enhances the policy’s robustness to unseen perceptual noise. We conduct extensive experiments both on quadruped and hexapod robots. Results demonstrate that ROLT is more robust than existing methods. Although trained in only a few dynamic settings, the learned policy generalizes well to multiple unseen dynamic conditions. Additionally, despite training with clean observations, the model handles challenging corruption noise during testing.

Details

UAI Conference 2025 Conference Paper

Letting Uncertainty Guide Your Multimodal Machine Translation

  • Wuyi Liu
  • Yue Gao
  • Yige Mao
  • Jing Zhao

Multimodal Machine Translation (MMT) leverages additional modalities, such as visual data, to enhance translation accuracy and resolve linguistic ambiguities inherent in text-only approaches. Recent advancements predominantly focus on integrating image information via attention mechanisms or feature fusion techniques. However, current approaches lack explicit mechanisms to quantify and manage the uncertainty during translation process, resulting in the utilization of image information being a black box. This makes it difficult to effectively address the issues of incomplete utilization of visual information and even potential degradation of translation quality when using visual information. To address these challenges, we introduce a novel Uncertainty-Guided Multimodal Machine Translation (UG-MMT) framework that redefines how translation systems handle ambiguity through systematic uncertainty reduction. Designed with plug-and-play flexibility, our framework enables seamless integration into existing MMT systems, requiring minimal modification while delivering significant performance gains.

Details

IJCAI Conference 2025 Conference Paper

Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance

  • Mingcheng Qu
  • Guang Yang
  • Donglin Di
  • Tonghua Su
  • Yue Gao
  • Yang Song
  • Lei Fan

Multimodal pathology-genomic analysis has become increasingly prominent in cancer survival prediction. However, existing studies mainly utilize multi-instance learning to aggregate patch-level features, neglecting the information loss of contextual and hierarchical details within pathology images. Furthermore, the disparity in data granularity and dimensionality between pathology and genomics leads to a significant modality imbalance. The high spatial resolution inherent in pathology data renders it a dominant role while overshadowing genomics in multimodal integration. In this paper, we propose a multimodal survival prediction framework that incorporates hypergraph learning to effectively capture both contextual and hierarchical details from pathology images. Moreover, it employs a modality rebalance mechanism and an interactive alignment fusion strategy to dynamically reweight the contributions of the two modalities, thereby mitigating the pathology-genomics imbalance. Quantitative and qualitative experiments are conducted on five TCGA datasets, demonstrating that our model outperforms advanced methods by over 3. 4% in C-Index performance. Code: https: //github. com/MCPathology/MRePath.

PDF Details DOI

EAAI Journal 2025 Journal Article

SAMGCN: A spatially-augmented multi-view graph convolutional network for identifying spatial domains

  • Hao Liu
  • Yue Gao
  • Ying-Lian Gao
  • Cui-Na Jiao
  • Junliang Shang
  • Jin-Xing Liu

Recent innovations in spatial transcriptomics have enabled the measurement of gene expression profiles while preserving the spatial organization of cells. This provides extensive opportunities to explore gene expression patterns in the tissue microenvironment. However, it remains a challenge to combine spatial information with gene expression to accurately identify spatial domains. In this study, a spatially-augmented multi-view graph convolutional network for identifying spatial domains (SAMGCN) is proposed. First, SAMGCN reconstructs gene expression data by incorporating spatial neighborhood information, which enhances gene expression features. It improves the quality of gene expression data and augments the characterization of spatial domains through the construction of spatial graphs, feature graphs, and spatial expression-weighted graphs. By extracting spatial information and gene expression data via convolutional operations, SAMGCN learns multi-view-specific embeddings and employs a contrastive strategy to refine and augment spatial neighborhood relationships, addressing limitations in spatial gene expression data. An attention mechanism is then employed to flexibly merge these embeddings, generating the final spot embedding. Additionally, a zero-inflated negative binomial decoder is used to capture the global probability distribution of gene expression profiles. Finally, the performance of SAMGCN has been validated across various platforms and spatial transcriptomics datasets of different scales, demonstrating its exceptional capability to process spatial transcriptomics data.

Details DOI

IROS Conference 2025 Conference Paper

SpongeBot: A Soft Magnetic Mini-Robot for Controlled Gastric Cell Sampling *

  • Jiyuan Tian
  • Nidhi Chhaparwal
  • Moonkwang Jeong
  • Yue Gao
  • Ann-Sophia Müller
  • Meng Zhang
  • Katharina Bosch
  • Karol Nowicki-Osuch

Early detection of gastrointestinal (GI) cancer is critical for improving treatment outcomes and survival rates. Yet conventional endoscopic techniques remain invasive and labor-intensive, thus presenting significant challenges for cancer screening on large populations. Current commercially available sponge-based sampling devices are passive and limited in their reach to the esophagus, hindering comprehensive sampling in the stomach. Here, for the first time, we report the SpongeBot – a non-invasive soft mini-robot designed for active cell sampling in the upper GI tract, with a particular focus on the stomach. The SpongeBot integrates an open-cell sponge and a magnetic actuator, enabling precise and controlled sampling under a wireless external magnetic field. To accommodate the intricate anatomy of the stomach, the robot is capable of transitioning between two modes of motion — the navigation and the sampling mode, allowing trajectory control and targeted sampling at desired locations. Kinematic model is established to accurately represent the locomotion of the robot on wet mucosa surfaces. Pilot testing on ex vivo porcine stomachs is successfully performed with sufficient cells sampled for subsequent clinical laboratory testing. Histological analysis shows the sampling causes no detectable damage to the mucosa layer. SpongeBot has the potential as a cell sampling device for the upper GI tract to be deployed in primary care settings for cancer prevention.

Details

EAAI Journal 2025 Journal Article

STDDAE: Identifying spatial domains in spatial transcriptomics by dual denoising autoencoder with attention mechanism

  • Yue Gao
  • Ying-Lian Gao
  • Cui-Na Jiao
  • Xu-Ran Dou
  • Feng Li
  • Jin-Xing Liu

Spatial transcriptomics provides a novel perspective for comprehending the intricate relationship between tissue structure and function, as well as for discovering new cell types and subtypes. However, it remains a significant challenge to accurately identify spatial domains with similar gene expression, which requires efficient combination of gene expression data, histology image information, and spatial location. To address this challenge, a novel dual denoising autoencoder with attention mechanism (STDDAE) is proposed. STDDAE integrates gene expression data, histology image information and spatial location, and the decoder consists of a master decoder and a follower decoder, which are jointly optimized to generate low-dimensional latent embeddings for precise spatial domain identification. The performance of STDDAE was evaluated across four datasets with varying resolutions and platforms. The experimental findings validated that STDDAE outperformed other cutting-edg methods in spatial domain identification, trajectory inference, and data denoising. Additionally, STDDAE successfully detected differentially expressed genes within identified spatial domains, which may be valuable in disease diagnosis, prognostic assessment, and treatment selection.

Details DOI

YNICL Journal 2025 Journal Article

The differential orbitofrontal activity and connectivity between atypical and typical major depressive disorder

  • Zhi-Peng Guo
  • Lei Chen
  • Li-Rong Tang
  • Yue Gao
  • Miao Qu
  • Lihong Wang
  • Chun-Hong Liu

OBJECTIVE: Atypical major depressive disorder (MDD) is a distinct subtype of MDD, characterized by increased appetite and/or weight gain, excessive sleep, leaden paralysis, and interpersonal rejection sensitivity. Delineating different neural circuits associated with atypical and typical MDD would better inform clinical personalized interventions. METHODS: Using resting-state fMRI, we investigated the voxel-level regional homogeneity (ReHo) and functional connectivity (FC) in 55 patients with atypical MDD, 51 patients with typical MDD, and 49 healthy controls (HCs). Support vector machine (SVM) approaches were applied to examine the validity of the findings in distinguishing the two types of MDD. RESULTS: Compared to patients with typical MDD and HCs, patients with atypical MDD had increased ReHo values in the right lateral orbitofrontal cortex (OFC) and enhanced FC between the right lateral OFC and right dorsolateral prefrontal cortex (dlPFC), and between the right striatum and left OFC. The ReHo in the right lateral OFC and the significant FCs found were significantly correlated with body mass index (BMI) in all groups of participants with MDD. The connectivity of the right striatum and left OFC was positively correlated with the retardation scores in the atypical MDD group. Using the ReHo of the right lateral OFC as a feature, we achieved 76.42% accuracy to differentiate atypical MDD from typical MDD. CONCLUSION: Our findings show that atypical MDD might be associated with altered OFC activity and connectivity. Furthermore, our findings highlight the key role of lateral OFC in atypical MDD, which may provide valuable information for future personalized interventions.

Details DOI

NeurIPS Conference 2024 Conference Paper

Assembly Fuzzy Representation on Hypergraph for Open-Set 3D Object Retrieval

  • Yang Xu
  • Yifan Feng
  • Jun Zhang
  • Jun-Hai Yong
  • Yue Gao

The lack of object-level labels presents a significant challenge for 3D object retrieval in the open-set environment. However, part-level shapes of objects often share commonalities across categories but remain underexploited in existing retrieval methods. In this paper, we introduce the Hypergraph-Based Assembly Fuzzy Representation (HARF) framework, which navigates the intricacies of open-set 3D object retrieval through a bottom-up lens of Part Assembly. To tackle the challenge of assembly isomorphism and unification, we propose the Hypergraph Isomorphism Convolution (HIConv) for smoothing and adopt the Isomorphic Assembly Embedding (IAE) module to generate assembly embeddings with geometric-semantic consistency. To address the challenge of open-set category generalization, our method employs high-order correlations and fuzzy representation to mitigate distribution skew through the Structure Fuzzy Reconstruction (SFR) module, by constructing a leveraged hypergraph based on local certainty and global uncertainty correlations. We construct three open-set retrieval datasets for 3D objects with part-level annotations: OP-SHNP, OP-INTRA, and OP-COSEG. Extensive experiments and ablation studies on these three benchmarks show our method outperforms current state-of-the-art methods.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition

  • Bingjun Luo
  • Haowen Wang
  • Jinpeng Wang
  • Junjie Zhu
  • Xibin Zhao
  • Yue Gao

With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents a more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt at deep NIR facial expression recognition and propose a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we construct a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks

  • Buqing Nie
  • Jingtian Ji
  • Yangqing Fu
  • Yue Gao

Deep Reinforcement Learning (DRL) has achieved remarkable advances in sequential decision tasks. However, recent works have revealed that DRL agents are susceptible to slight perturbations in observations. This vulnerability raises concerns regarding the effectiveness and robustness of deploying such agents in real-world applications. In this work, we propose a novel robust reinforcement learning method called SortRL, which improves the robustness of DRL policies against observation perturbations from the perspective of the network architecture. We employ a novel architecture for the policy network that incorporates global $l_\infty$ Lipschitz continuity and provide a convenient method to enhance policy robustness based on the output margin. Besides, a training framework is designed for SortRL, which solves given tasks while maintaining robustness against $l_\infty$ bounded perturbations on the observations. Several experiments are conducted to evaluate the effectiveness of our method, including classic control tasks and video games. The results demonstrate that SortRL achieves state-of-the-art robustness performance against different perturbation strength.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition

  • Bingjun Luo
  • Zewen Wang
  • Jinpeng Wang
  • Junjie Zhu
  • Xibin Zhao
  • Yue Gao

Illumination variation has been a long-term challenge in real-world facial expression recognition (FER). Under uncontrolled or non-visible light conditions, near-infrared (NIR) can provide a simple and alternative solution to obtain high-quality images and supplement the geometric and texture details that are missing in the visible (VIS) domain. Due to the lack of large-scale NIR facial expression datasets, directly extending VIS FER methods to the NIR spectrum may be ineffective. Additionally, previous heterogeneous image synthesis methods are restricted by low controllability without prior task knowledge. To tackle these issues, we present the first approach, called for NIR-FER Stochastic Differential Equations (NFER-SDE), that transforms face expression appearance between heterogeneous modalities to the overfitting problem on small-scale NIR data. NFER-SDE can take the whole VIS source image as input and, together with domain-specific knowledge, guide the preservation of modality-invariant information in the high-frequency content of the image. Extensive experiments and ablation studies show that NFER-SDE significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Multi-scale Consistency for Robust 3D Registration via Hierarchical Sinkhorn Tree

  • Chengwei Ren
  • Yifan Feng
  • Weixiang Zhang
  • Xiao-Ping Zhang
  • Yue Gao

We study the problem of retrieving accurate correspondence through multi-scale consistency (MSC) for robust point cloud registration. Existing works in a coarse-to-fine manner either suffer from severe noisy correspondences caused by unreliable coarse matching or struggle to form outlier-free coarse-level correspondence sets. To tackle this, we present Hierarchical Sinkhorn Tree (HST), a pruned tree structure designed to hierarchically measure the local consistency of each coarse correspondence across multiple feature scales, thereby filtering out the local dissimilar ones. In this way, we convert the modeling of MSC for each correspondence into a BFS traversal with pruning of a K-ary tree rooted at the superpoint, with its K nearest neighbors in the feature pyramid serving as child nodes. To achieve efficient pruning and accurate vicinity characterization, we further propose a novel overlap-aware Sinkhorn Distance, which retains only the most likely overlapping points for local measurement and next level exploration. The modeling process essentially involves traversing a pair of HSTs synchronously and aggregating the consistency measures of corresponding tree nodes. Extensive experiments demonstrate HST consistently outperforms the state-of-the-art methods on both indoor and outdoor benchmarks.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Negative Prompt Driven Complementary Parallel Representation for Open-World 3D Object Retrieval

  • Yang Xu
  • Yifan Feng
  • Yue Gao

The limited availability of supervised labels (positive information) poses a notable challenge for open-world retrieval. However, negative information is more easily obtained but remains underexploited in current methods. In this paper, we introduce the Negative Prompt Driven Complementary Parallel Representation (NPCP) framework, which navigates the complexities of open-world retrieval through the lens of Negative Prompts. Specifically, we employ the Parallel Exclusive Embedding (PEE) to effectively utilize the prompt information, bilaterally capturing both explicit negative and implicit positive signals. To address the challenges of embedding unification and generalization, our method leverages high-order correlations among objects through the Complementary Structure Tuning (CST), by constructing a complementary hypergraph based on bi-directional and cross-category correlations. We have developed four multimodal datasets for open-world 3D object retrieval with negative prompts: NPMN, NPAB, NPNT, and NPES. Extensive experiments and ablation studies on these four benchmarks demonstrate the superiority of our method over current state-of-the-art approaches.

PDF Details DOI

ICML Conference 2024 Conference Paper

Position: Topological Deep Learning is the New Frontier for Relational Learning

  • Theodore Papamarkou
  • Tolga Birdal
  • Michael M. Bronstein
  • Gunnar E. Carlsson
  • Justin Curry
  • Yue Gao
  • Mustafa Hajij
  • Roland Kwitt

Topological deep learning (TDL) is a rapidly evolving field that uses topological features to understand and design deep learning models. This paper posits that TDL is the new frontier for relational learning. TDL may complement graph representation learning and geometric deep learning by incorporating topological concepts, and can thus provide a natural choice for various machine learning settings. To this end, this paper discusses open problems in TDL, ranging from practical benefits to theoretical foundations. For each problem, it outlines potential solutions and future research opportunities. At the same time, this paper serves as an invitation to the scientific community to actively participate in TDL research to unlock the potential of this emerging field.

Details

NeurIPS Conference 2024 Conference Paper

Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph

  • Yang Xu
  • Yifan Feng
  • Jun Zhang
  • Jun-Hai Yong
  • Yue Gao

Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets, leading to contradictory optimization for superposed categories. In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for this task. Specifically, we propose the Hierarchical Retrace Embedding (HRE) module to overcome the global disequilibrium of unseen categories by fully leveraging the multi-level category information. Besides, tackling the feature overlap and class confusion problem, we perform the Structured Equilibrium Tuning (SET) module to utilize more equilibrial correlations among objects and generalize to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we generate four semi-open 3DOR datasets with multi-level labels for benchmarking. Results demonstrate that the proposed method can effectively generate the hierarchical embeddings of 3D objects and generalize them towards semi-open environments.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction

  • Yangqing Fu
  • Ming Sun
  • Buqing Nie
  • Yue Gao

Monte Carlo Tree Search (MCTS) algorithms such as AlphaGo and MuZero have achieved superhuman performance in many challenging tasks. However, the computational complexity of MCTS-based algorithms is influenced by the size of the search space. To address this issue, we propose a novel probability tree state abstraction (PTSA) algorithm to improve the search efficiency of MCTS. A general tree state abstraction with path transitivity is defined. In addition, the probability tree state abstraction is proposed for fewer mistakes during the aggregation step. Furthermore, the theoretical guarantees of the transitivity and aggregation error bound are justified. To evaluate the effectiveness of the PTSA algorithm, we integrate it with state-of-the-art MCTS-based algorithms, such as Sampled MuZero and Gumbel MuZero. Experimental results on different tasks demonstrate that our method can accelerate the training process of state-of-the-art algorithms with 10%-45% search space reduction.

PDF Details

AAAI Conference 2023 System Paper

EasyRec: An Easy-to-Use, Extendable and Efficient Framework for Building Industrial Recommendation Systems

  • Mengli Cheng
  • Yue Gao
  • Guoqiang Liu
  • HongSheng Jin

We present EasyRec, an easy-to-use, extendable and efficient recommendation framework for building industrial recommendation systems. Our EasyRec framework is superior in the following aspects:first, EasyRec adopts a modular and pluggable design pattern to reduce the efforts to build custom models; second, EasyRec implements hyper-parameter optimization and feature selection algorithms to improve model performance automatically; third, EasyRec applies online learning to adapt to the ever-changing data distribution. The code is released: https://github.com/alibaba/EasyRec.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Learning Deep Hierarchical Features with Spatial Regularization for One-Class Facial Expression Recognition

  • Bingjun Luo
  • Junjie Zhu
  • Tianyu Yang
  • Sicheng Zhao
  • Chao Hu
  • Xibin Zhao
  • Yue Gao

Existing methods on facial expression recognition (FER) are mainly trained in the setting when multi-class data is available. However, to detect the alien expressions that are absent during training, this type of methods cannot work. To address this problem, we develop a Hierarchical Spatial One Class Facial Expression Recognition Network (HS-OCFER) which can construct the decision boundary of a given expression class (called normal class) by training on only one-class data. Specifically, HS-OCFER consists of three novel components. First, hierarchical bottleneck modules are proposed to enrich the representation power of the model and extract detailed feature hierarchy from different levels. Second, multi-scale spatial regularization with facial geometric information is employed to guide the feature extraction towards emotional facial representations and prevent the model from overfitting extraneous disturbing factors. Third, compact intra-class variation is adopted to separate the normal class from alien classes in the decision space. Extensive evaluations on 4 typical FER datasets from both laboratory and wild scenarios show that our method consistently outperforms state-of-the-art One-Class Classification (OCC) approaches.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

NeuralGF: Unsupervised Point Normal Estimation by Learning Neural Gradient Function

  • Qing Li
  • Huifang Feng
  • Kanle Shi
  • Yue Gao
  • Yi Fang
  • Yu-Shen Liu
  • Zhizhong Han

Normal estimation for 3D point clouds is a fundamental task in 3D geometry processing. The state-of-the-art methods rely on priors of fitting local surfaces learned from normal supervision. However, normal supervision in benchmarks comes from synthetic shapes and is usually not available from real scans, thereby limiting the learned priors of these methods. In addition, normal orientation consistency across shapes remains difficult to achieve without a separate post-processing procedure. To resolve these issues, we propose a novel method for estimating oriented normals directly from point clouds without using ground truth normals as supervision. We achieve this by introducing a new paradigm for learning neural gradient functions, which encourages the neural network to fit the input point clouds and yield unit-norm gradients at the points. Specifically, we introduce loss functions to facilitate query points to iteratively reach the moving targets and aggregate onto the approximated surface, thereby learning a global surface representation of the data. Meanwhile, we incorporate gradients into the surface approximation to measure the minimum signed deviation of queries, resulting in a consistent gradient field associated with the surface. These techniques lead to our deep unsupervised oriented normal estimator that is robust to noise, outliers and density variations. Our excellent results on widely used benchmarks demonstrate that our method can learn more accurate normals for both unoriented and oriented normal estimation tasks than the latest methods. The source code and pre-trained model are publicly available.

PDF Details

NeurIPS Conference 2022 Conference Paper

Grow and Merge: A Unified Framework for Continuous Categories Discovery

  • Xinwei Zhang
  • Jianwen Jiang
  • Yutong Feng
  • Zhi-Fan Wu
  • Xibin Zhao
  • Hai Wan
  • Mingqian Tang
  • Rong Jin

Although a number of studies are devoted to novel category discovery, most of them assume a static setting where both labeled and unlabeled data are given at once for finding new categories. In this work, we focus on the application scenarios where unlabeled data are continuously fed into the category discovery system. We refer to it as the {\bf Continuous Category Discovery} ({\bf CCD}) problem, which is significantly more challenging than the static setting. A common challenge faced by novel category discovery is that different sets of features are needed for classification and category discovery: class discriminative features are preferred for classification, while rich and diverse features are more suitable for new category mining. This challenge becomes more severe for dynamic setting as the system is asked to deliver good performance for known classes over time, and at the same time continuously discover new classes from unlabeled data. To address this challenge, we develop a framework of {\bf Grow and Merge} ({\bf GM}) that works by alternating between a growing phase and a merge phase: in the growing phase, it increases the diversity of features through a continuous self-supervised learning for effective category mining, and in the merging phase, it merges the grown model with a static one to ensure satisfying performance for known classes. Our extensive studies verify that the proposed GM framework is significantly more effective than the state-of-the-art approaches for continuous category discovery.

PDF Details

JBHI Journal 2022 Journal Article

Hematoma Expansion Context Guided Intracranial Hemorrhage Segmentation and Uncertainty Estimation

  • Xiangyu Li
  • Gongning Luo
  • Wei Wang
  • Kuanquan Wang
  • Yue Gao
  • Shuo Li

Accurate segmentation of the Intracranial Hemorrhage (ICH) in non-contrast CT images is significant for computer-aided diagnosis. Although existing methods have achieved remarkable 1 1 The code will be available from https://github.com/JohnleeHIT/SLEX-Net.results, none of them incorporated ICH’s prior information in their methods. In this work, for the first time, we proposed a novel SLice EXpansion Network (SLEX-Net), which incorporated hematoma expansion in the segmentation architecture by directly modeling the hematoma variation among adjacent slices. Firstly, a new module named Slice Expansion Module (SEM) was built, which can effectively transfer contextual information between two adjacent slices by mapping predictions from one slice to another. Secondly, to perceive contextual information from both upper and lower slices, we designed two information transmission paths: forward and backward slice expansion, and aggregated results from those paths with a novel weighing strategy. By further exploiting intra-slice and inter-slice context with the information paths, the network significantly improved the accuracy and continuity of segmentation results. Moreover, the proposed SLEX-Net enables us to conduct an uncertainty estimation with one-time inference, which is much more efficient than existing methods. We evaluated the proposed SLEX-Net and compared it with some state-of-the-art methods. Experimental results demonstrate that our method makes significant improvements in all metrics on segmentation performance and outperforms other existing uncertainty estimation methods in terms of several metrics.

Details DOI

ICML Conference 2022 Conference Paper

Lazy Estimation of Variable Importance for Large Neural Networks

  • Yue Gao
  • Abby Stevens
  • Garvesh Raskutti
  • Rebecca Willett

As opaque predictive models increasingly impact many areas of modern life, interest in quantifying the importance of a given input variable for making a specific prediction has grown. Recently, there has been a proliferation of model-agnostic methods to measure variable importance (VI) that analyze the difference in predictive power between a full model trained on all variables and a reduced model that excludes the variable(s) of interest. A bottleneck common to these methods is the estimation of the reduced model for each variable (or subset of variables), which is an expensive process that often does not come with theoretical guarantees. In this work, we propose a fast and flexible method for approximating the reduced model with important inferential guarantees. We replace the need for fully retraining a wide neural network by a linearization initialized at the full model parameters. By adding a ridge-like penalty to make the problem convex, we prove that when the ridge penalty parameter is sufficiently large, our method estimates the variable importance measure with an error rate of O(1/n) where n is the number of training samples. We also show that our estimator is asymptotically normal, enabling us to provide confidence bounds for the VI estimates. We demonstrate through simulations that our method is fast and accurate under several data-generating regimes, and we demonstrate its real-world applicability on a seasonal climate forecasting example.

Details

NeurIPS Conference 2022 Conference Paper

On the Limitations of Stochastic Pre-processing Defenses

  • Yue Gao
  • I Shumailov
  • Kassem Fawaz
  • Nicolas Papernot

Defending against adversarial examples remains an open problem. A common belief is that randomness at inference increases the cost of finding adversarial inputs. An example of such a defense is to apply a random transformation to inputs prior to feeding them to the model. In this paper, we empirically and theoretically investigate such stochastic pre-processing defenses and demonstrate that they are flawed. First, we show that most stochastic defenses are weaker than previously thought; they lack sufficient randomness to withstand even standard attacks like projected gradient descent. This casts doubt on a long-held assumption that stochastic defenses invalidate attacks designed to evade deterministic defenses and force attackers to integrate the Expectation over Transformation (EOT) concept. Second, we show that stochastic defenses confront a trade-off between adversarial robustness and model invariance; they become less effective as the defended model acquires more invariance to their randomization. Future work will need to decouple these two effects. We also discuss implications and guidance for future research.

PDF Details

NeurIPS Conference 2021 Conference Paper

Aligning Pretraining for Detection via Object-Level Contrastive Learning

  • Fangyun Wei
  • Yue Gao
  • Zhirong Wu
  • Han Hu
  • Stephen Lin

Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning. Such generality for transfer learning, however, sacrifices specificity if we are interested in a certain downstream task. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. In this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-level representations are introduced via selective search bounding boxes as object proposals; 2) the pretraining network architecture incorporates the same dedicated modules used in the detection pipeline (e. g. FPN); 3) the pretraining is equipped with object detection properties such as object-level translation invariance and scale invariance. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-art results for transfer performance on COCO detection using a Mask R-CNN framework. Code is available at https: //github. com/hologerry/SoCo.

PDF Details

AAAI Conference 2021 Conference Paper

Domain General Face Forgery Detection by Learning to Weight

  • Ke Sun
  • Hong Liu
  • Qixiang Ye
  • Yue Gao
  • Jianzhuang Liu
  • Ling Shao
  • Rongrong Ji

In this paper, we propose a domain-general model, termed learning-to-weight (LTW), that guarantees face detection performance across multiple domains, particularly the target domains that are never seen before. However, various face forgery methods cause complex and biased data distributions, making it challenging to detect fake faces in unseen domains. We argue that different faces contribute differently to a detection model trained on multiple domains, making the model likely to fit domain-specific biases. As such, we propose the LTW approach based on the meta-weight learning algorithm, which configures different weights for face images from different domains. The LTW network can balance the model’s generalizability across multiple domains. Then, the meta-optimization calibrates the source domain’s gradient enabling more discriminative features to be learned. The detection ability of the network is further improved by introducing an intra-class compact loss. Extensive experiments on several commonly used deepfake datasets to demonstrate the effectiveness of our method in detecting synthetic faces. Code and supplemental material are available at https: //github. com/skJack/LTW.

PDF Details

AAAI Conference 2021 Conference Paper

Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network

  • Jiayi Ji
  • Yunpeng Luo
  • Xiaoshuai Sun
  • Fuhai Chen
  • Gen Luo
  • Yongjian Wu
  • Yue Gao
  • Rongrong Ji

Transformer-based architectures have shown great success in image captioning, where object regions are encoded and then attended into the vectorial representations to guide the caption decoding. However, such vectorial representations only contain region-level information without considering the global information reflecting the entire image, which fails to expand the capability of complex multi-modal reasoning in image captioning. In this paper, we introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation, and then adaptively guide the decoder to generate high-quality captions. In GET, a Global Enhanced Encoder is designed for the embedding of the global feature, and a Global Adaptive Decoder are designed for the guidance of the caption generation. The former models intra- and inter-layer global representation by taking advantage of the proposed Global Enhanced Attention and a layer-wise fusion module. The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation. Extensive experiments on MS COCO dataset demonstrate the superiority of our GET over many state-of-the-arts.

PDF Details

ICML Conference 2021 Conference Paper

Leveraging Non-uniformity in First-order Non-convex Optimization

  • Jincheng Mei
  • Yue Gao
  • Bo Dai 0001
  • Csaba Szepesvári
  • Dale Schuurmans

Classical global convergence results for first-order methods rely on uniform smoothness and the Ł{}ojasiewicz inequality. Motivated by properties of objective functions that arise in machine learning, we propose a non-uniform refinement of these notions, leading to \emph{Non-uniform Smoothness} (NS) and \emph{Non-uniform Ł{}ojasiewicz inequality} (NŁ{}). The new definitions inspire new geometry-aware first-order methods that are able to converge to global optimality faster than the classical $\Omega(1/t^2)$ lower bounds. To illustrate the power of these geometry-aware methods and their corresponding non-uniform analysis, we consider two important problems in machine learning: policy gradient optimization in reinforcement learning (PG), and generalized linear model training in supervised learning (GLM). For PG, we find that normalizing the gradient ascent method can accelerate convergence to $O(e^{- c \cdot t})$ (where $c > 0$) while incurring less overhead than existing algorithms. For GLM, we show that geometry-aware normalized gradient descent can also achieve a linear convergence rate, which significantly improves the best known results. We additionally show that the proposed geometry-aware gradient descent methods escape landscape plateaus faster than standard gradient descent. Experimental results are used to illustrate and complement the theoretical findings.

Details

AAAI Conference 2020 Conference Paper

Attention-Based Multi-Modal Fusion Network for Semantic Scene Completion

  • Siqi Li
  • Changqing Zou
  • Yipeng Li
  • Xibin Zhao
  • Yue Gao

This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task of inferring the occupancy and semantic labels of a volumetric 3D scene from single-view RGB-D images. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously via leveraging the experience of inferring 2D semantic segmentation from RGB-D images as well as the reliable depth cues in spatial dimension. It is achieved by employing a multi-modal fusion architecture boosted from 2D semantic segmentation and a 3D semantic completion network empowered by residual attention blocks. We validate our method on both the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results show that our method respectively achieves the gains of 2. 5% and 2. 6% on the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the state-of-the-art method.

PDF Details

AAAI Conference 2020 Conference Paper

Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering

  • Jianwen Jiang
  • Ziqiang Chen
  • Haojie Lin
  • Xibin Zhao
  • Yue Gao

Understanding questions and finding clues for answers are the key for video question answering. Compared with image question answering, video question answering (Video QA) requires to find the clues accurately on both spatial and temporal dimension simultaneously, and thus is more challenging. However, the relationship between spatio-temporal information and question still has not been well utilized in most existing methods for Video QA. To tackle this problem, we propose a Question-Guided Spatio-Temporal Contextual Attention Network (QueST) method. In QueST, we divide the semantic features generated from question into two separate parts: the spatial part and the temporal part, respectively guiding the process of constructing the contextual attention on spatial and temporal dimension. Under the guidance of the corresponding contextual attention, visual features can be better exploited on both spatial and temporal dimensions. To evaluate the effectiveness of the proposed method, experiments are conducted on TGIF-QA dataset, MSRVTT- QA dataset and MSVD-QA dataset. Experimental results and comparisons with the state-of-the-art methods have shown that our method can achieve superior performance.

PDF Details

AAAI Conference 2020 Conference Paper

Hypergraph Label Propagation Network

  • Yubo Zhang
  • Nan Wang
  • Yufeng Chen
  • Changqing Zou
  • Hai Wan
  • Xinbin Zhao
  • Yue Gao

In recent years, with the explosion of information on the Internet, there has been a large amount of data produced, and analyzing these data is useful and has been widely employed in real world applications. Since data labeling is costly, lots of research has focused on how to efficiently label data through semi-supervised learning. Among the methods, graph and hypergraph based label propagation algorithms have been a widely used method. However, traditional hypergraph learning methods may suffer from their high computational cost. In this paper, we propose a Hypergraph Label Propagation Network (HLPN) which combines hypergraphbased label propagation and deep neural networks in order to optimize the feature embedding for optimal hypergraph learning through an end-to-end architecture. The proposed method is more effective and also efficient for data labeling compared with traditional hypergraph learning methods. We verify the effectiveness of our proposed HLPN method on a real-world microblog dataset gathered from Sina Weibo. Experiments demonstrate that the proposed method can significantly outperform the state-of-the-art methods and alternative approaches.

PDF Details

AAAI Conference 2019 Conference Paper

DeepCCFV: Camera Constraint-Free Multi-View Convolutional Neural Network for 3D Object Retrieval

  • Zhengyue Huang
  • Zhehui Zhao
  • Hengguang Zhou
  • Xibin Zhao
  • Yue Gao

3D object retrieval has a compelling demand in the field of computer vision with the rapid development of 3D vision technology and increasing applications of 3D objects. 3D objects can be described in different ways such as voxel, point cloud, and multi-view. Among them, multi-view based approaches proposed in recent years show promising results. Most of them require a fixed predefined camera position setting which provides a complete and uniform sampling of views for objects in the training stage. However, this causes heavy over-fitting problems which make the models failed to generalize well in free camera setting applications, particularly when insufficient views are provided. Experiments show the performance drastically drops when the number of views reduces, hindering these methods from practical applications. In this paper, we investigate the over-fitting issue and remove the constraint of the camera setting. First, two basic feature augmentation strategies Dropout and Dropview are introduced to solve the over-fitting issue, and a more precise and more efficient method named DropMax is proposed after analyzing the drawback of the basic ones. Then, by reducing the over-fitting issue, a camera constraint-free multi-view convolutional neural network named DeepCCFV is constructed. Extensive experiments on both single-modal and cross-modal cases demonstrate the effectiveness of the proposed method in free camera settings comparing with existing state-of-theart 3D object retrieval methods.

PDF Details

IJCAI Conference 2019 Conference Paper

Dynamic Hypergraph Neural Networks

  • Jianwen Jiang
  • Yuxuan Wei
  • Yifan Feng
  • Jingxuan Cao
  • Yue Gao

In recent years, graph/hypergraph-based deep learning methods have attracted much attention from researchers. These deep learning methods take graph/hypergraph structure as prior knowledge in the model. However, hidden and important relations are not directly represented in the inherent structure. To tackle this issue, we propose a dynamic hypergraph neural networks framework (DHGNN), which is composed of the stacked layers of two modules: dynamic hypergraph construction (DHG) and hypergrpah convolution (HGC). Considering initially constructed hypergraph is probably not a suitable representation for data, the DHG module dynamically updates hypergraph structure on each layer. Then hypergraph convolution is introduced to encode high-order data relations in a hypergraph structure. The HGC module includes two phases: vertex convolution and hyperedge convolution, which are designed to aggregate feature among vertices and hyperedges, respectively. We have evaluated our method on standard datasets, the Cora citation network and Microblog dataset. Our method outperforms state-of-the-art methods. More experiments are conducted to demonstrate the effectiveness and robustness of our method to diverse data distributions.

PDF Details

AAAI Conference 2019 Conference Paper

Hypergraph Neural Networks

  • Yifan Feng
  • Haoxuan You
  • Zizhao Zhang
  • Rongrong Ji
  • Yue Gao

In this paper, we present a hypergraph neural networks (HGNN) framework for data representation learning, which can encode high-order data correlation in a hypergraph structure. Confronting the challenges of learning representation for complex data in real practice, we propose to incorporate such data structure in a hypergraph, which is more flexible on data modeling, especially when dealing with complex data. In this method, a hyperedge convolution operation is designed to handle the data correlation during representation learning. In this way, traditional hypergraph learning procedure can be conducted using hyperedge convolution operations efficiently. HGNN is able to learn the hidden layer representation considering the high-order data structure, which is a general framework considering the complex data correlations. We have conducted experiments on citation network classification and visual object recognition tasks and compared HGNN with graph convolutional networks and other traditional methods. Experimental results demonstrate that the proposed HGNN method outperforms recent state-of-theart methods. We can also reveal from the results that the proposed HGNN is superior when dealing with multi-modal data compared with existing methods.

PDF Details

AAAI Conference 2019 Conference Paper

MeshNet: Mesh Neural Network for 3D Shape Representation

  • Yutong Feng
  • Yifan Feng
  • Haoxuan You
  • Xibin Zhao
  • Yue Gao

Mesh is an important and powerful type of data for 3D shapes and widely studied in the field of computer vision and computer graphics. Regarding the task of 3D shape representation, there have been extensive research efforts concentrating on how to represent 3D shapes well using volumetric grid, multi-view and point cloud. However, there is little effort on using mesh data in recent years, due to the complexity and irregularity of mesh data. In this paper, we propose a mesh neural network, named MeshNet, to learn 3D shape representation from mesh data. In this method, face-unit and feature splitting are introduced, and a general architecture with available and effective blocks are proposed. In this way, MeshNet is able to solve the complexity and irregularity problem of mesh and conduct 3D shape representation well. We have applied the proposed MeshNet method in the applications of 3D shape classification and retrieval. Experimental results and comparisons with the state-of-the-art methods demonstrate that the proposed MeshNet can achieve satisfying 3D shape classification and retrieval performance, which indicates the effectiveness of the proposed method on 3D shape representation.

PDF Details

AAAI Conference 2019 Conference Paper

MLVCNN: Multi-Loop-View Convolutional Neural Network for 3D Shape Retrieval

  • Jianwen Jiang
  • Di Bao
  • Ziqiang Chen
  • Xibin Zhao
  • Yue Gao

3D shape retrieval has attracted much attention and become a hot topic in computer vision field recently. With the development of deep learning, 3D shape retrieval has also made great progress and many view-based methods have been introduced in recent years. However, how to represent 3D shapes better is still a challenging problem. At the same time, the intrinsic hierarchical associations among views still have not been well utilized. In order to tackle these problems, in this paper, we propose a multi-loop-view convolutional neural network (MLVCNN) framework for 3D shape retrieval. In this method, multiple groups of views are extracted from different loop directions first. Given these multiple loop views, the proposed MLVCNN framework introduces a hierarchical view-loop-shape architecture, i. e. , the view level, the loop level, and the shape level, to conduct 3D shape representation from different scales. In the view-level, a convolutional neural network is first trained to extract view features. Then, the proposed Loop Normalization and LSTM are utilized for each loop of view to generate the loop-level features, which considering the intrinsic associations of the different views in the same loop. Finally, all the loop-level descriptors are combined into a shape-level descriptor for 3D shape representation, which is used for 3D shape retrieval. Our proposed method has been evaluated on the public 3D shape benchmark, i. e. , ModelNet40. Experiments and comparisons with the state-of-the-art methods show that the proposed MLVCNN method can achieve significant performance improvement on 3D shape retrieval tasks. Our MLVCNN outperforms the state-of-the-art methods by the mAP of 4. 84% in 3D shape retrieval task. We have also evaluated the performance of the proposed method on the 3D shape classification task where MLVCNN also achieves superior performance compared with recent methods.

PDF Details

AAAI Conference 2019 Conference Paper

PVRNet: Point-View Relation Neural Network for 3D Shape Recognition

  • Haoxuan You
  • Yifan Feng
  • Xibin Zhao
  • Changqing Zou
  • Rongrong Ji
  • Yue Gao

Three-dimensional (3D) shape recognition has drawn much research attention in the field of computer vision. The advances of deep learning encourage various deep models for 3D feature representation. For point cloud and multi-view data, two popular 3D data modalities, different models are proposed with remarkable performance. However the relation between point cloud and views has been rarely investigated. In this paper, we introduce Point-View Relation Network (PVRNet), an effective network designed to well fuse the view features and the point cloud feature with a proposed relation score module. More specifically, based on the relation score module, the point-single-view fusion feature is first extracted by fusing the point cloud feature and each single view feature with point-singe-view relation, then the pointmulti-view fusion feature is extracted by fusing the point cloud feature and the features of different number of views with point-multi-view relation. Finally, the point-single-view fusion feature and point-multi-view fusion feature are further combined together to achieve a unified representation for a 3D shape. Our proposed PVRNet has been evaluated on ModelNet40 dataset for 3D shape classification and retrieval. Experimental results indicate our model can achieve significant performance improvement compared with the state-of-the-art models.

PDF Details

IJCAI Conference 2018 Conference Paper

Dynamic Hypergraph Structure Learning

  • Zizhao Zhang
  • Haojie Lin
  • Yue Gao

In recent years, hypergraph modeling has shown its superiority on correlation formulation among samples and has wide applications in classification, retrieval, and other tasks. In all these works, the performance of hypergraph learning highly depends on the generated hypergraph structure. A good hypergraph structure can represent the data correlation better, and vice versa. Although hypergraph learning has attracted much attention recently, most of existing works still rely on a static hypergraph structure, and little effort concentrates on optimizing the hypergraph structure during the learning process. To tackle this problem, we propose a dynamic hypergraph structure learning method in this paper. In this method, given the originally generated hypergraph structure, the objective of our work is to simultaneously optimize the label projection matrix (the common task in hypergraph learning) and the hypergraph structure itself. More specifically, in this formulation, the label projection matrix is related to the hypergraph structure, and the hypergraph structure is associated with the data correlation from both the label space and the feature space. Here, we alternatively learn the optimal label projection matrix and the hypergraph structure, leading to a dynamic hypergraph structure during the learning process. We have applied the proposed method in the tasks of 3D shape recognition and gesture recognition. Experimental results on 4 public datasets show better performance compared with the state-of-the-art methods. We note that the proposed method can be further applied in other tasks.

PDF Details

AAAI Conference 2018 Conference Paper

EMD Metric Learning

  • Zizhao Zhang
  • Yubo Zhang
  • Xibin Zhao
  • Yue Gao

Earth Mover’s Distance (EMD), targeting at measuring the many-to-many distances, has shown its superiority and been widely applied in computer vision tasks, such as object recognition, hyperspectral image classification and gesture recognition. However, there is still little effort concentrated on optimizing the EMD metric towards better matching performance. To tackle this issue, we propose an EMD metric learning algorithm in this paper. In our method, the objective is to learn a discriminative distance metric for EMD ground distance matrix generation which can better measure the similarity between compared subjects. More specifically, given a group of labeled data from different categories, we first select a subset of training data and then optimize the metric for ground distance matrix generation. Here, both the EMD metric and the EMD flow-network are alternatively optimized until a steady EMD value can be achieved. This method is able to generate a discriminative ground distance matrix which can further improve the EMD distance measurement. We then apply our EMD metric learning method on two tasks, i. e. , multiview object classification and document classification. The experimental results have shown better performance of our proposed EMD metric learning method compared with the traditional EMD method and the state-of-the-art methods. It is noted that the proposed EMD metric learning method can be also used in other applications.

PDF Details

AAAI Conference 2018 Conference Paper

Energy-Efficient Automatic Train Driving by Learning Driving Patterns

  • Jin Huang
  • Yue Gao
  • Sha Lu
  • Xibin Zhao
  • Yangdong Deng
  • Ming Gu

Railway is regarded as the most sustainable means of modern transportation. With the fast-growing of fleet size and the railway mileage, the energy consumption of trains is becoming a serious concern globally. The nature of railway offers a unique opportunity to optimize the energy efficiency of locomotives by taking advantage of the undulating terrains along a route. The derivation of an energy-optimal train driving solution, however, proves to be a significant challenge due to the high dimension, nonlinearity, complex constraints, and timevarying characteristic of the problem. An optimized solution can only be attained by considering both the complex environmental conditions of a given route and the inherent characteristics of a locomotive. To tackle the problem, this paper employs a high-order correlation learning method for online generation of the energy optimized train driving solutions. Based on the driving data of experienced human drivers, a hypergraph model is used to learn the optimal embedding from the specified features for the decision of a driving operation. First, we design a feature set capturing the driving status. Next all the training data are formulated as a hypergraph and an inductive learning process is conducted to obtain the embedding matrix. The hypergraph model can be used for real-time generation of driving operation. We also proposed a reinforcement updating scheme, which offers the capability of sustainable enhancement on the hypergraph model in industrial applications. The learned model can be used to determine an optimized driving operation in real-time tested on the Hardware-in-Loop platform. Validation experiments proved that the energy consumption of the proposed solution is around 10% lower than that of average human drivers.

PDF Details

AAAI Conference 2018 Conference Paper

Hypergraph Learning With Cost Interval Optimization

  • Xibin Zhao
  • Nan Wang
  • Heyuan Shi
  • Hai Wan
  • Jin Huang
  • Yue Gao

In many classification tasks, the misclassification costs of different categories usually vary significantly. Under such circumstances, it is essential to identify the importance of different categories and thus assign different misclassification losses in many applications, such as medical diagnosis, saliency detection and software defect prediction. However, we note that it is infeasible to determine the accurate cost value without great domain knowledge. In most common cases, we may just have the information that which category is more important than the other categories, i. e. , the identification of defect-prone softwares is more important than that of defect-free. To tackle these issues, in this paper, we propose a hypergraph learning method with cost interval optimization, which is able to handle cost interval when data is formulated using the high-order relationships. In this way, data correlations are modeled by a hypergraph structure, which has the merit to exploit the underlying relationships behind the data. With a cost-sensitive hypergraph structure, in order to improve the performance of the classifier without precise cost value, we further introduce cost interval optimization to hypergraph learning. In this process, the optimization on cost interval achieves better performance instead of choosing uncertain fixed cost in the learning process. To evaluate the effectiveness of the proposed method, we have conducted experiments on two groups of dataset, i. e. , the NASA Metrics Data Program (NASA) dataset and UCI Machine Learning Repository (UCI) dataset. Experimental results and comparisons with state-of-the-art methods have exhibited better performance of our proposed method.

PDF Details

IJCAI Conference 2018 Conference Paper

Iterative Metric Learning for Imbalance Data Classification

  • Nan Wang
  • Xibin Zhao
  • Yu Jiang
  • Yue Gao

In many classification applications, the amount of data from different categories usually vary significantly, such as software defect predication and medical diagnosis. Under such circumstances, it is essential to propose a proper method to solve the imbalance issue among the data. However, most of the existing methods mainly focus on improving the performance of classifiers rather than searching for an appropriate way to find an effective data space for classification. In this paper, we propose a method named Iterative Metric Learning (IML) to explore the correlations among imbalance data and construct an effective data space for classification. Given the imbalance training data, it is important to select a subset of training samples for each testing data. Thus, we aim to find a more stable neighborhood for testing data using the iterative metric learning strategy. To evaluate the effectiveness of the proposed method, we have conducted experiments on two groups of dataset, i. e. , the NASA Metrics Data Program (NASA) dataset and UCI Machine Learning Repository (UCI) dataset. Experimental results and comparisons with state-of-the-art methods have exhibited better performance of our proposed method.

PDF Details

JBHI Journal 2018 Journal Article

Multi-Hypergraph Learning for Incomplete Multimodality Data

  • Mingxia Liu
  • Yue Gao
  • Pew-Thian Yap
  • Dinggang Shen

Multi-modality data convey complementary information that can be used to improve the accuracy of prediction models in disease diagnosis. However, effectively integrating multi-modality data remains a challenging problem, especially when the data are incomplete. For instance, more than half of the subjects in the Alzheimer's disease neuroimaging initiative (ADNI) database have no fluorodeoxyglucose positron emission tomography and cerebrospinal fluid data. Currently, there are two commonly used strategies to handle the problem of incomplete data: 1) discard samples having missing features; and 2) impute those missing values via specific techniques. In the first case, a significant amount of useful information is lost and, in the second case, additional noise and artifacts might be introduced into the data. Also, previous studies generally focus on the pairwise relationships among subjects, without considering their underlying complex (e. g. , high-order) relationships. To address these issues, in this paper, we propose a multi-hypergraph learning method for dealing with incomplete multimodality data. Specifically, we first construct multiple hypergraphs to represent the high-order relationships among subjects by dividing them into several groups according to the availability of their data modalities. A hypergraph regularized transductive learning method is then applied to these groups for automatic diagnosis of brain diseases. Extensive evaluation of the proposed method using all subjects in the baseline ADNI database indicates that our method achieves promising results in AD/MCI classification, compared with the state-of-the-art methods.

Details DOI

IJCAI Conference 2018 Conference Paper

Personality-Aware Personalized Emotion Recognition from Physiological Signals

  • Sicheng Zhao
  • Guiguang Ding
  • Jungong Han
  • Yue Gao

Emotion recognition methodologies from physiological signals are increasingly becoming personalized, due to the subjective responses of different subjects to physical stimuli. Existing works mainly focused on modelling the involved physiological corpus of each subject, without considering the psychological factors. The latent correlation among different subjects has also been rarely examined. We propose to investigate the influence of personality on emotional behavior in a hypergraph learning framework. Assuming that each vertex is a compound tuple (subject, stimuli), multi-modal hypergraphs can be constructed based on the personality correlation among different subjects and on the physiological correlation among corresponding stimuli. To reveal the different importance of vertices, hyperedges, and modalities, we assign each of them with weights. The emotion relevance learned on the vertex-weighted multi-modal multi-task hypergraphs is employed for emotion recognition. We carry out extensive experiments on the ASCERTAIN dataset and the results demonstrate the superiority of the proposed method.

PDF Details

IJCAI Conference 2018 Conference Paper

Robust Face Sketch Synthesis via Generative Adversarial Fusion of Priors and Parametric Sigmoid

  • Shengchuan Zhang
  • Rongrong Ji
  • Jie Hu
  • Yue Gao
  • Chia-Wen Lin

Despite the extensive progress in face sketch synthesis, existing methods are mostly workable under constrained conditions, such as fixed illumination, pose, background and ethnic origin that are hardly to control in real-world scenarios. The key issue lies in the difficulty to use data under fixed conditions to train a model against imaging variations. In this paper, we propose a novel generative adversarial network termed pGAN, which can generate face sketches efficiently using training data under fixed conditions and handle the aforementioned uncontrolled conditions. In pGAN, we embed key photo priors into the process of synthesis and design a parametric sigmoid activation function for compensating illumination variations. Compared to the existing methods, we quantitatively demonstrate that the proposed method can work well on face photos in the wild.

PDF Details

YNIMG Journal 2018 Journal Article

Structural correlates of literacy difficulties in the second language: Evidence from Mandarin-speaking children learning English

  • Hehui Li
  • James R. Booth
  • Nathalie N. Bélanger
  • Xiaoxia Feng
  • Mengyu Tian
  • Weiyi Xie
  • Manli Zhang
  • Yue Gao

Several neuroimaging studies have explored the neural basis of literacy difficulties in the second language (L2). However, it remains unclear whether the associated neural alterations are related to literacy abilities in the first language (L1). Using magnetic resonance imaging, we explore this issue with two experiments in Mandarin-speaking children learning English as second language. In the first experiment, we investigated children with literacy difficulties in L2 and L1 (poor in both, PB) and children with literacy difficulties only in L2 (poor in English, PE). We compared the brain structure in these two groups to a control literacy (CL) group. The results showed that the CL group had significantly less gray matter volume in the left supramarginal gyrus compared to the PB group and moderately less gray matter volume compared to the PE group. In addition, the PB group had significant greater gray matter volume in the left medial fusiform gyrus compared to the PE group and had marginally greater gray matter volume compared to the CL group. In the second experiment, we explored the relationship between the two atypical regions and literacy abilities in the two languages in an independent sample consisting of children with typical literacy. Correlation analyses revealed that the left supramarginal gyrus was significantly associated with literacy performance only in the second language, English, whereas the left medial fusiform gyrus did not correlate with the performances in either L1 or L2. Taken together, these findings suggest that literacy difficulties in an alphabetic L2 are associated with a structural abnormality in the left supramarginal gyrus, a region implicated in phonological processing, which is independent of literacy abilities in the native language.

Details DOI

AAAI Conference 2017 Conference Paper

Active Learning with Cross-Class Similarity Transfer

  • Yuchen Guo
  • Guiguang Ding
  • Yue Gao
  • Jungong Han

How to save labeling efforts for training supervised classi- fiers is an important research topic in machine learning community. Active learning (AL) and transfer learning (TL) are two useful tools to achieve this goal, and their combination, i. e. , transfer active learning (T-AL) has also attracted considerable research interest. However, existing T-AL approaches consider to transfer knowledge from a source/auxiliary domain which has the same class labels as the target domain, but ignore the relationship among classes. In this paper, we investigate a more practical setting where the classes in source domain are related/similar to but different from the target domain classes. Specifically, we propose a novel cross-class T-AL approach to simultaneously transfer knowledge from source domain and actively annotate the most informative samples in target domain so that we can train satisfactory classifiers with as few labeled samples as possible. In particular, based on the class-class similarity and sample-sample similarity, we adopt a similarity propagation to find the source domain samples that can well capture the characteristics of a target class and then transfer the similar samples as the (pseudo) labeled data for the target class. In turn, the labeled and transferred samples are used to train classifiers and actively select new samples for annotation. Extensive experiments on three datasets demonstrate that the proposed approach outperforms significantly the state-of-the-art related approaches.

PDF Details

IJCAI Conference 2017 Conference Paper

Approximating Discrete Probability Distribution of Image Emotions by Multi-Modal Features Fusion

  • Sicheng Zhao
  • Guiguang Ding
  • Yue Gao
  • Jungong Han

Existing works on image emotion recognition mainly assigned the dominant emotion category or average dimension values to an image based on the assumption that viewers can reach a consensus on the emotion of images. However, the image emotions perceived by viewers are subjective by nature and highly related to the personal and situational factors. On the other hand, image emotions can be conveyed by different features, such as semantics and aesthetics. In this paper, we propose a novel machine learning approach that formulates the categorical image emotions as a discrete probability distribution (DPD). To associate emotions with the extracted visual features, we present a weighted multi-modal shared sparse leaning to learn the combination coefficients, with which the DPD of an unseen image can be predicted by linearly integrating the DPDs of the training images. The representation abilities of different modalities are jointly explored and the optimal weight of each modality is automatically learned. Extensive experiments on three datasets verify the superiority of the proposed method, as compared to the state-of-the-art.

PDF Details

TIST Journal 2017 Journal Article

Event Classification in Microblogs via Social Tracking

  • Yue Gao
  • Hanwang Zhang
  • Xibin Zhao
  • Shuicheng Yan

Social media websites have become important information sharing platforms. The rapid development of social media platforms has led to increasingly large-scale social media data, which has shown remarkable societal and marketing values. There are needs to extract important events in live social media streams. However, microblogs event classification is challenging due to two facts, i.e., the short/conversational nature and the incompatible meanings between the text and the corresponding image in social posts, and the rapidly evolving contents. In this article, we propose to conduct event classification via deep learning and social tracking. First, we introduce a Multi-modal Multi-instance Deep Network (M 2 DN) for microblogs classification, which is able to handle the weakly labeled microblogs data oriented from the incompatible meanings inside microblogs. Besides predicting each microblogs as predefined events, we propose to employ social tracking to extract social-related auxiliary information to enrich the testing samples. We extract a set of candidate-relevant microblogs in a short time window by using social connections, such as related users and geographical locations. All these selected microblogs and the testing data are formulated in a Markov Random Field model. The inference on the Markov Random Field is conducted to update the classification results of the testing microblogs. This method is evaluated on the Brand-Social-Net dataset for classification of 20 events. Experimental results and comparison with the state of the arts show that the proposed method can achieve better performance for the event classification task.

Details DOI

IJCAI Conference 2017 Conference Paper

SitNet: Discrete Similarity Transfer Network for Zero-shot Hashing

  • Yuchen Guo
  • Guiguang Ding
  • Jungong Han
  • Yue Gao

Hashing has been widely utilized for fast image retrieval recently. With semantic information as supervision, hashing approaches perform much better, especially when combined with deep convolution neural network(CNN). However, in practice, new concepts emerge every day, making collecting supervised information for re-training hashing model infeasible. In this paper, we propose a novel zero-shot hashing approach, called Discrete Similarity Transfer Network (SitNet), to preserve the semantic similarity between images from both ``seen'' concepts and new ``unseen'' concepts. Motivated by zero-shot learning, the semantic vectors of concepts are adopted to capture the similarity structures among classes, making the model trained with seen concepts generalize well for unseen ones benefiting from the transferability of the semantic vector space. We adopt a multi-task architecture to exploit the supervised information for seen concepts and the semantic vectors simultaneously. Moreover, a discrete hashing layer is integrated into the network for hashcode generating to avoid the information loss caused by real-value relaxation in training phase, which is a critical problem in existing works. Experiments on three benchmarks validate the superiority of SitNet to the state-of-the-arts.

PDF Details

IJCAI Conference 2017 Conference Paper

Synthesizing Samples for Zero-shot Learning

  • Yuchen Guo
  • Guiguang Ding
  • Jungong Han
  • Yue Gao

Zero-shot learning (ZSL) is to construct recognition models for unseen target classes that have no labeled samples for training. It utilizes the class attributes or semantic vectors as side information and transfers supervision information from related source classes with abundant labeled samples. Existing ZSL approaches adopt an intermediary embedding space to measure the similarity between a sample and the attributes of a target class to perform zero-shot classification. However, this way may suffer from the information loss caused by the embedding process and the similarity measure cannot fully make use of the data distribution. In this paper, we propose a novel approach which turns the ZSL problem into a conventional supervised learning problem by synthesizing samples for the unseen classes. Firstly, the probability distribution of an unseen class is estimated by using the knowledge from seen classes and the class attributes. Secondly, the samples are synthesized based on the distribution for the unseen class. Finally, we can train any supervised classifiers based on the synthesized samples. Extensive experiments on benchmarks demonstrate the superiority of the proposed approach to the state-of-the-art ZSL approaches.

PDF Details

IJCAI Conference 2017 Conference Paper

TUCH: Turning Cross-view Hashing into Single-view Hashing via Generative Adversarial Nets

  • Xin Zhao
  • Guiguang Ding
  • Yuchen Guo
  • Jungong Han
  • Yue Gao

Cross-view retrieval, which focuses on searching images as response to text queries or vice versa, has received increasing attention recently. Cross-view hashing is to efficiently solve the cross-view retrieval problem with binary hash codes. Most existing works on cross-view hashing exploit multi-view embedding method to tackle this problem, which inevitably causes the information loss in both image and text domains. Inspired by the Generative Adversarial Nets (GANs), this paper presents a new model that is able to Turn Cross-view Hashing into single-view hashing (TUCH), thus enabling the information of image to be preserved as much as possible. TUCH is a novel deep architecture that integrates a language model network T for text feature extraction, a generator network G to generate fake images from text feature and a hashing network H for learning hashing functions to generate compact binary codes. Our architecture effectively unifies joint generative adversarial learning and cross-view hashing. Extensive empirical evidence shows that our TUCH approach achieves state-of-the-art results, especially on text to image retrieval, based on image-sentences datasets, i. e. standard IAPRTC-12 and large-scale Microsoft COCO.

PDF Details

IJCAI Conference 2017 Conference Paper

Vertex-Weighted Hypergraph Learning for Multi-View Object Classification

  • Lifan Su
  • Yue Gao
  • Xibin Zhao
  • Hai Wan
  • Ming Gu
  • Jiaguang Sun

3D object classification with multi-view representation has become very popular, thanks to the progress on computer techniques and graphic hardware, and attracted much research attention in recent years. Regarding this task, there are mainly two challenging issues, i. e. , the complex correlation among multiple views and the possible imbalance data issue. In this work, we propose to employ the hypergraph structure to formulate the relationship among 3D objects, taking the advantage of hypergraph on high-order correlation modelling. However, traditional hypergraph learning method may suffer from the imbalance data issue. To this end, we propose a vertex-weighted hypergraph learning algorithm for multi-view 3D object classification, introducing an updated hypergraph structure. In our method, the correlation among different objects is formulated in a hypergraph structure and each object (vertex) is associated with a corresponding weight, weighting the importance of each sample in the learning process. The learning process is conducted on the vertex-weighted hypergraph and the estimated object relevance is employed for object classification. The proposed method has been evaluated on two public benchmarks, i. e. , the NTU and the PSB datasets. Experimental results and comparison with the state-of-the-art methods and recent deep learning method demonstrate the effectiveness of our proposed method.

PDF Details

AAAI Conference 2017 Conference Paper

Zero-Shot Recognition via Direct Classifier Learning with Transferred Samples and Pseudo Labels

  • Yuchen Guo
  • Guiguang Ding
  • Jungong Han
  • Yue Gao

As an interesting and emerging topic, zero-shot recognition (ZSR) makes it possible to train a recognition model by specifying the category’s attributes when there are no labeled exemplars available. The fundamental idea for ZSR is to transfer knowledge from the abundant labeled data in different but related source classes via the class attributes. Conventional ZSR approaches adopt a two-step strategy in test stage, where the samples are projected into the attribute space in the first step, and then the recognition is carried out based on considering the relationship between samples and classes in the attribute space. Due to this intermediate transformation, information loss is unavoidable, thus degrading the performance of the overall system. Rather than following this two-step strategy, in this paper, we propose a novel one-step approach that is able to perform ZSR in the original feature space by using directly trained classifiers. To tackle the problem that no labeled samples of target classes are available, we propose to assign pseudo labels to samples based on the reliability and diversity, which in turn will be used to train the classi- fiers. Moreover, we adopt a robust SVM that accounts for the unreliability of pseudo labels. Extensive experiments on four datasets demonstrate consistent performance gains of our approach over the state-of-the-art two-step ZSR approaches.

PDF Details

IJCAI Conference 2016 Conference Paper

Semi-Supervised Active Learning with Cross-Class Sample Transfer

  • Yuchen Guo
  • Guiguang Ding
  • Yue Gao
  • Jianmin Wang

To save the labeling efforts for training a classification model, we can simultaneously adopt Active Learning (AL) to select the most informative samples for human labeling, and Semi-supervised Learning (SSL) to construct effective classifiers using a few labeled samples and a large number of unlabeled samples. Recently, using Transfer Learning (TL) to enhance AL and SSL, i. e. , T-SS-AL, has gained considerable attention. However, existing T-SS-AL methods mostly focus on the situation where the source domain and the target domain share the same classes. In this paper, we consider a more practical and challenging setting where the source domain and the target domain have different but related classes. We propose a novel cross-class sample transfer based T-SS-AL method, called CC-SS-AL, to exploit the information from the source domain. Our key idea is to select samples from the source domain which are very similar to the target domain classes and assign pseudo labels to them for classifier training. Extensive experiments on three datasets verify the efficacy of the proposed method.

PDF Details

TIST Journal 2015 Journal Article

Depth Error Elimination for RGB-D Cameras

  • Yue Gao
  • You Yang
  • Yi Zhen
  • Qionghai Dai

The rapid spreading of RGB-D cameras has led to wide applications of 3D videos in both academia and industry, such as 3D entertainment and 3D visual understanding. Under these circumstances, extensive research efforts have been dedicated to RGB-D camera--oriented topics. In these topics, quality promotion of depth videos with the temporal characteristic is emerging and important. Due to the limited exposure time of RGB-D cameras, object movement can easily lead to motion blurs in intensive images, which can further result in obvious artifacts (holes or fake boundaries) in the corresponding depth frames. With regard to this problem, we propose a depth error elimination method based on time series analysis to remove the artifacts in depth images. In this method, we first locate the regions with erroneous depths in intensive images by using motion blur detection based on a time series analysis model. This is based on the fact that the depth image is calculated by intensive color images that are captured synchronously by RGB-D cameras. Then, the artifacts, such as holes or fake boundaries, are fixed by a depth error elimination method. To evaluate the performance of the proposed method, we conducted experiments on 250 images. Experimental results demonstrate that the proposed method can locate the error regions correctly and eliminate these artifacts effectively. The quality of depth video can be improved significantly by using the proposed method.

Details DOI

TIST Journal 2015 Journal Article

When Location Meets Social Multimedia

  • Rongrong Ji
  • Yue Gao
  • Wei Liu
  • Xing Xie
  • Qi Tian
  • Xuelong Li

Coming with the popularity of multimedia sharing platforms such as Facebook and Flickr, recent years have witnessed an explosive growth of geographical tags on social multimedia content. This trend enables a wide variety of emerging applications, for example, mobile location search, landmark recognition, scene reconstruction, and touristic recommendation, which range from purely research prototype to commercial systems. In this article, we give a comprehensive survey on these applications, covering recent advances in recognition and mining of geographical-aware social multimedia. We review related work in the past decade regarding to location recognition, scene summarization, tourism suggestion, 3D building modeling, mobile visual search and city navigation. At the end, we further discuss potential challenges, future topics, as well as open issues related to geo-social multimedia computing, recognition, mining, and analytics.

Details DOI
v2026.09.13