Arrow Research search

Author name cluster

Hao Tang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

62 papers
2 author rows

Possible papers

62

AAAI Conference 2026 Conference Paper

Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

  • Hao Tang
  • Yu Liu
  • Shuanglin Yan
  • Fei Shen
  • Shengfeng He
  • Jing Qin

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Fine-Grained Image Retrieval via Dual-Vision Adaptation

  • Xin Jiang
  • Meiqi Cao
  • Hao Tang
  • Fei Shen
  • Zechao Li

Fine-Grained Image Retrieval~(FGIR) faces challenges in learning discriminative visual representations to retrieve images with similar fine-grained features. Current leading FGIR solutions typically follow two regimes: enforce pairwise similarity constraints in the semantic embedding space, or incorporate a localization sub-network to fine-tune the entire model. However, such two regimes tend to overfit the training data while forgetting the knowledge gained from large-scale pre-training, thus reducing their generalization ability. In this paper, we propose a Dual-Vision Adaptation (DVA) approach for FGIR, which guides the frozen pre-trained model to perform FGIR through collaborative sample and feature adaptation. Specifically, we design Object-Perceptual Adaptation, which modifies input samples to help the pre-trained model perceive critical objects and elements within objects that are helpful for category prediction. Meanwhile, we propose In-Context Adaptation, which introduces a small set of parameters for feature adaptation without modifying the pre-trained parameters. This makes the FGIR task using these adapted features closer to the task solved during the pre-training. Additionally, to balance retrieval efficiency and performance, we propose Discrimination Perception Transfer to transfer the discriminative knowledge in the object-perceptual adaptation to the image encoder using the knowledge distillation mechanism. Extensive experiments show that DVA performs well on three fine-grained datasets.

PDF Details DOI

AAAI Conference 2026 Conference Paper

FourierPET: Deep Fourier-based Unrolled Network for Low-count PET Reconstruction

  • Zheng Zhang
  • Hao Tang
  • Yingying Hu
  • Zhanli Hu
  • Jing Qin

Low-count positron emission tomography (PET) reconstruction is a challenging inverse problem due to severe degradations arising from Poisson noise, photon scarcity, and attenuation correction errors. Existing deep learning methods typically address these in the spatial domain with an undifferentiated optimization objective, making it difficult to disentangle overlapping artifacts and limiting correction effectiveness. In this work, we perform a Fourier-domain analysis and reveal that these degradations are spectrally separable: Poisson noise and photon scarcity cause high-frequency phase perturbations, while attenuation errors suppress low-frequency amplitude components. Leveraging this insight, we propose FourierPET, a Fourier-based unrolled reconstruction framework grounded in the Alternating Direction Method of Multipliers. It consists of three tailored modules: a spectral consistency module that enforces global frequency alignment to maintain data fidelity, an amplitude–phase correction module that decouples and compensates for high-frequency phase distortions and low-frequency amplitude suppression, and a dual adjustment module that accelerates convergence during iterative reconstruction. Extensive experiments demonstrate that FourierPET achieves state-of-the-art performance with significantly fewer parameters, while offering enhanced interpretability through frequency-aware correction.

PDF Details DOI

JBHI Journal 2026 Journal Article

HemoSC-P: A Hemodynamic Semantic Channel Paradigm for Cardiovascular Parameter Estimation

  • Xi Qiu
  • Zhifei Zhang
  • Hailin Cao
  • Li Yang
  • Hao Tang
  • Lianyang Zhang

As a leading cause of death worldwide, cardiovascular disease demands more precise monitoring and early warning systems, posing significant challenges to modern healthcare. However, cardiovascular early warning systems often face two major dilemmas: the “black box dilemma” leads to unreliable estimation results due to limited physiological interpretability, and limited robustness across subjects and blood pressure states arises from physiological heterogeneity. This study innovatively maps the methods of semantic extraction and channel modeling to the cardiovascular system, inspired by the 6G network concept of transmitting meaning rather than data under the semantic communication paradigm. It proposes a novel hemodynamic channel-guided cardiovascular parameter estimation paradigm (HemoSC-P). This paradigm adopts a dual-pillar modeling framework: the semantic pillar utilizes a multi-scale convolutional and phase-aware attention architecture to model the non-stationary dynamics of cardiovascular signals, elevating feature alignment from the temporal domain to the physiological domain. The cardiovascular channel, parameterized by the Windkessel circuit model, serves dual roles as a semantic pathway from cardiac contractions to observable physiological signals and as a repository of physiological knowledge. It guides channel-level physiologically deformable attention to achieve inverse estimation of cardiovascular parameters. To validate this paradigm, this study employs non-invasive blood pressure estimation as a representative case study. Validation was conducted on three representative public datasets (UCI-BP, MIMIC-III, PPG-BP). On MIMIC-III, the mean absolute error $\pm$ standard deviation for systolic and diastolic blood pressure reached 3. 04 $\pm$ 3. 24 mmHg and 2. 57 $\pm$ 2. 70 mmHg, respectively. Multi-dataset validation indicates that this paradigm surpasses benchmark methods in accuracy and stability while maintaining scalability.

Details DOI

AAAI Conference 2026 Conference Paper

ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

  • Yihua Shao
  • Xiaofeng Lin
  • Xinwei Long
  • Siyu Chen
  • Minxi Yan
  • Yang Liu
  • Ziyang Yan
  • Ao Ma

Enabling multi-task adaptation in pre-trained Low-Rank Adaptation (LoRA) models is crucial for enhancing their generalization capabilities. Most existing pre-trained LoRA fusion methods decompose weight matrices, sharing similar parameters, while fusion divergent ones. However, this paradigm inevitably induces inter-weight conflicts and leads to catastrophic domain forgetting. While incremental learning enables adaptation to multiple tasks, it struggles to achieve generalization in few-shot scenarios. Consequently, when the weight data follows a long-tailed distribution, it can lead to forgetting in the fused weights. To address this issue, we propose In-Context Meta LoRA Fusion (ICM-Fusion), a novel framework that synergizes meta-learning with in-context adaptation. The key innovation lies in our task vector arithmetic, which dynamically balances conflicting optimization directions across domains through learned manifold projections. ICM-Fusion obtains the optimal task vector orientation for the fused model in the latent space by adjusting the orientation of the task vectors. Subsequently, the fused LoRA is reconstructed by a self-designed Fusion VAE (F-VAE) to realize multi-task LoRA generation. We have conducted extensive experiments on visual and linguistic tasks, and the experimental results demonstrate that ICM-Fusion can be adapted to a wide range of architectural models and applied to various tasks. Compared to the current pre-trained LoRA fusion method, ICM-Fusion fused LoRA can significantly reduce the multi-tasking loss and can even achieve task enhancement in few-shot scenarios.

PDF Details DOI

AAAI Conference 2026 Conference Paper

IMAGGarment+: Efficient Attribute-Wise Diffusion for Garment Generation

  • Jian Yu
  • Fei Shen
  • Cong Wang
  • Yanpeng Sun
  • Hao Tang
  • Qin Guo
  • Xiaoyu Du

Diffusion models have advanced fine-grained garment generation, yet balancing controllability, efficiency, and texture fidelity remains challenging. Adapter-based methods often yield incoherent details, while full fine-tuning is computationally expensive and prone to overwriting pretrained priors. To address these limitations, we propose IMAGGarment+, an efficient diffusion framework for controllable and high-quality garment synthesis. It comprises two key modules designed for efficient and attribute-aware conditioning. First, we introduce an attribute-wise feature extractor (AFE) that disentangles key garment attributes, silhouette, logo, position, and color, into parallel latent streams. Each stream is optimized independently via LoRA, ensuring minimal parameter overhead while retaining expressive capacity. Second, we develop an attribute-adaptive attention (AA) module to inject attribute-specific cues into the generative process through a selective, layer-wise injection strategy. Specifically, silhouette and color features are injected into early decoder layers to guide structural and appearance formation, while logo features are propagated across all layers to ensure cross-scale consistency. Extensive experiments on fine-grained garment benchmarks demonstrate that IMAGGarment+ outperforms state-of-the-art baselines with less than 20% additional parameters, validating its effectiveness and efficiency.

PDF Details DOI

AAAI Conference 2026 Conference Paper

MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging

  • Shufeng Kong
  • Zijie Wang
  • Nuan Cui
  • Hao Tang
  • Yihan Meng
  • Yuanyuan Wei
  • Feifan Chen
  • Yingheng Wang

Automated interpretation of medical images demands robust modeling of complex visual-semantic relationships while addressing annotation scarcity, label imbalance, and clinical plausibility constraints. We introduce MIRNet (Medical Image Reasoner Network), a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning. Tongue image diagnosis is a particularly challenging domain that requires fine-grained visual and semantic understanding. Our approach leverages self-supervised masked autoencoder (MAE) to learn transferable visual representations from unlabeled data; employs graph attention networks (GAT) to model label correlations through expert-defined structured graphs; enforces clinical priors via constraint-aware optimization using KL divergence and regularization losses; and mitigates imbalance using asymmetric loss (ASL) and boosting ensembles. To address annotation scarcity, we also introduce TongueAtlas-4K, a comprehensive expert-curated benchmark comprising 4,000 images annotated with 22 diagnostic labels–representing the largest public dataset in tongue analysis. Validation shows our method achieves state-of-the-art performance. While optimized for tongue diagnosis, the framework readily generalizes to broader diagnostic medical imaging tasks.

PDF Details DOI

AAMAS Conference 2026 Conference Paper

Resolving Task Objective Conflicts in Unified Model via Task-Aware Mixture-of-Experts

  • Jiaxing Zhang
  • Hao Tang

Recently, multimodal understanding (MMU) and text-to-image generation (T2I) have been integrated into a single autoregressive (AR) architecture, achieving initial unification. However, existing works focus on representation-level studies and overlook potential conflicts in AR architectures’ internal information flow during training different tasks. Motivated by this gap, we identify a deeper issue, Task Objective Conflict (TOC), arising from AR architectures’ internal information flow, which causes negative transfer and catastrophic forgetting when training MMU and T2I jointly. To address this issue, we proposed UniDecouple, which decouples internal modules for different tasks to construct task-specific optimization subpaths. To implement UniDecouple, we employ a Task-Aware Mixture of Experts (TA-MoE), comprising Hierarchical Expert Routing and Hybrid Expert Collaboration, trained in two stages: first to build task-specific experts, then jointly fine-tuned to balance specialization and overall coordination. Extensive experiments on both understanding and generation benchmarks demonstrate that UniDecouple preserves strong understanding ability while achieving generation quality comparable to state-of-the-art methods, offering a new perspective for unified modeling.

PDF

AAMAS Conference 2026 Conference Paper

Structured Agent Distillation for Large Language Model Agents

  • Jun Liu
  • Zhenglun Kong
  • Peiyan Dong
  • Changdi Yang
  • Tianqin Li
  • Yanyue Xie
  • Yifan Gong
  • Xuan Shen

Largelanguagemodels(LLMs)exhibitstrongcapabilitiesasdecisionmaking agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical deployment is constrained by high inference costs and large model sizes. We propose Structured Agent Distillation, the first framework to distill a ReAct-based LLM agent into a smaller model while preserving both reasoning fidelity and action consistency. Our method introduces a structured, span-level distillation strategy that explicitly segments trajectories into reasoning and action spans, enabling fine-grained alignment beyond standard token-level imitation. Unlike other advanced distillation methods, Our method segments trajectories into [REASON] and [ACT] spans, applying segmentspecific losses to align each component with the teacher’s behavior. This structure-aware supervision enables compact agents to better Corresponding Authors: Pu Zhao, Hao Tang. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). https: //doi. org/10. 65109/OLHJ8062 replicate the teacher’s decision process. Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines, achieving significant compression with minimal performance drop. Scaling and ablation results further highlight the importance of span-level alignment for efficient and deployable agents. We will release code upon acceptance.

PDF

AAAI Conference 2026 Conference Paper

TR-DQ: Time-Rotation Diffusion Quantization

  • Yihua Shao
  • Deyang Lin
  • Minxi Yan
  • Siyu Chen
  • Fanhu Zeng
  • Minwen Liao
  • Ao Ma
  • Ziyang Yan

Diffusion models have been widely adopted in image and video generation. However, their complex network architecture leads to high inference overhead for its generation process. Existing diffusion quantization methods primarily focus on the quantization of the model structure while ignoring the impact of time-steps variation during sampling. At the same time, most current approaches fail to account for significant activations that cannot be eliminated, resulting in substantial performance degradation after quantization. To address these issues, we propose Time-Rotation Diffusion Quantization (TR-DQ), a novel quantization method incorporating time-step and rotation-based optimization. TR-DQ first divides the sampling process based on time-steps and applies a rotation matrix to smooth activations and weights dynamically. For different time-steps, a dedicated hyperparameter is introduced for adaptive timing modeling, which enables dynamic quantization across different time steps. Additionally, we also explore the compression potential of Classifier-Free Guidance (CFG-wise) to establish a foundation for subsequent work. TR-DQ achieves state-of-the-art (SOTA) performance on image generation and video generation tasks and a 1.38-1.89× speedup and 1.97-2.58× memory reduction in inference compared to existing quantization methods.

PDF Details DOI

AAAI Conference 2025 Conference Paper

A Training-free Synthetic Data Selection Method for Semantic Segmentation

  • Hao Tang
  • Siyue Yu
  • Jian Pang
  • Bingfeng Zhang

Training semantic segmenter with synthetic data has been attracting great attention due to its easy accessibility and huge quantities. Most previous methods focused on producing large-scale synthetic image-annotation samples and then training the segmenter with all of them. However, such a solution remains a main challenge in that the poor-quality samples are unavoidable, and using them to train the model will damage the training process. In this paper, we propose a training-free Synthetic Data Selection (SDS) strategy with CLIP to select high-quality samples for building a reliable synthetic dataset. Specifically, given massive synthetic image-annotation pairs, we first design a Perturbation-based CLIP Similarity (PCS) to measure the reliability of synthetic image, thus removing samples with low-quality images. Then we propose a class-balance Annotation Similarity Filter (ASF) by comparing the synthetic annotation with the response of CLIP to remove the samples related to low-quality annotations. The experimental results show that using our method significantly reduces the data size by half, while the trained segmenter achieves higher performance.

PDF Details DOI

AAAI Conference 2025 Conference Paper

ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling

  • Jianan Jiang
  • Hao Tang
  • Zhilin Jiang
  • Weiren Yu
  • Di Wu

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims to minimize the distance between sketches and corresponding images in the embedding space. However, scalability is hindered by the growing complexity of solutions, mainly due to the abstract nature of fine-grained sketches. In this paper, we propose an effective approach to narrow the gap between the two domains. It mainly facilitates unified mutual information sharing both intra- and inter-samples, rather than treating them as a single feature alignment problem between modalities. Specifically, our approach includes: (i) Employing dual weight-sharing networks to optimize alignment within the sketch and image domain, which also effectively mitigates model learning saturation issues. (ii) Introducing an objective optimization function based on contrastive loss to enhance the model's ability to align features in both intra- and inter-samples. (iii) Presenting a self-supervised Multi-Scale Token Recycling (MSTR) Module featured by recycling discarded patch tokens in multi-scale features, further enhancing representation capability and retrieval performance. Our framework achieves excellent results on CNN- and ViT-based backbones. Extensive experiments demonstrate its superiority over existing methods. We also introduce Cloths-V1, the first professional fashion sketch-image dataset, utilized to validate our method and will be beneficial for other applications.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Boosting Adversarial Transferability with Spatial Adversarial Alignment

  • Zhaoyu Chen
  • HaiJing Guo
  • Kaixun Jiang
  • Jiyuan Fu
  • Xinyu Zhou
  • Dingkang Yang
  • Hao Tang
  • Bo Li

Deep neural networks are vulnerable to adversarial examples that exhibit transferability across various models. Numerous approaches are proposed to enhance the transferability of adversarial examples, including advanced optimization, data augmentation, and model modifications. However, these methods still show limited transferability, partiovovocularly in cross-architecture scenarios, such as from CNN to ViT. To achieve high transferability, we propose a technique termed Spatial Adversarial Alignment (SAA), which employs an alignment loss and leverages a witness model to fine-tune the surrogate model. Specifically, SAA consists of two key parts: spatial-aware alignment and adversarial-aware alignment. First, we minimize the divergences of features between the two models in both global and local regions, facilitating spatial alignment. Second, we introduce a self-adversarial strategy that leverages adversarial examples to impose further constraints, aligning features from an adversarial perspective. Through this alignment, the surrogate model is trained to concentrate on the common features extracted by the witness model. This facilitates adversarial attacks on these shared features, thereby yielding perturbations that exhibit enhanced transferability. Extensive experiments on various architectures on ImageNet show that aligned surrogate models based on SAA can provide higher transferable adversarial examples, especially in cross-architecture attacks.

PDF Details

IJCAI Conference 2025 Conference Paper

Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning

  • Hao Tang
  • Shengfeng He
  • Jing Qin

Few-shot learning (FSL) addresses the challenge of classifying novel classes with limited training samples. While some methods leverage semantic knowledge from smaller-scale models to mitigate data scarcity, these approaches often introduce noise and bias due to the data’s inherent simplicity. In this paper, we propose a novel framework, Synergistic Knowledge Transfer (SynTrans), which effectively transfers diverse and complementary knowledge from large multimodal models to empower the off-the-shelf few-shot learner. Specifically, SynTrans employs CLIP as a robust teacher and uses a few-shot vision encoder as a weak student, distilling semantic-aligned visual knowledge via an unsupervised proxy task. Subsequently, a training-free synergistic knowledge mining module facilitates collaboration among large multimodal models to extract high-quality semantic knowledge. Building upon this, a visual-semantic bridging module enables bi-directional knowledge transfer between visual and semantic spaces, transforming explicit visual and implicit semantic knowledge into category-specific classifier weights. Finally, SynTrans introduces a visual weight generator and a semantic weight reconstructor to adaptively construct optimal multimodal FSL classifiers. Experimental results on four FSL datasets demonstrate that SynTrans, even when paired with a simple few-shot vision encoder, significantly outperforms current state-of-the-art methods.

PDF Details DOI

EAAI Journal 2025 Journal Article

Digital twin-driven reinforcement learning-based operational management for customized manufacturing

  • Hao Tang
  • Minghao Cheng
  • Uzair Aslam Bhatti
  • Bo Xu
  • Nan Zhou
  • Rong Guo
  • Bing Wei

Due to the increasing complexity of customer demands for different batches and types of products, manufacturing operations management has been facing the challenge of uncertain product arrival times and resource processing times in customized manufacturing (CM). This paper proposes a dynamic scheduling method to solve the uncertainty in CM via the integration of the digital twin and fuzzy reinforcement learning methods. In this study, a digital twin-driven framework is first designed to describe the operation management system (OMS) hierarchies. Then a semi-Markov decision process (MDP) model with fuzzy definition is built by abstracting the stochastic scheduling process. To solve the semi-MDP model, an asynchronous multi-edge co-training method is presented to train a fuzzy deep neural network through closed-loop control of virtual commissioning, illustrating how the digital twin-driven OMS adapts to dynamic production requirements. Finally, the proposed method is verified by the performance of comparative experiment. Experimental results show that for randomly arriving products, the proposed method guarantees timely training and scheduling decisions and has the highest total system profit compared to other competing methods (Hybrid Multi-Agent System Negotiation and Ant Colony Optimization (HMA), Onto_MDP, and Deep Q Networks (DQN)). Also, the proposed method shows better scheduling performance in terms of average decision time, average training time and number of finished products when resources are abnormal.

Details DOI

NeurIPS Conference 2025 Conference Paper

Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment

  • Kaixun Jiang
  • Zhaoyu Chen
  • HaiJing Guo
  • Jinglun Li
  • Jiyuan Fu
  • Pinxue Guo
  • Hao Tang
  • Bo Li

Preference alignment in diffusion models has primarily focused on benign human preferences (e. g. , aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e. g. , reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective.

PDF Details

IJCAI Conference 2025 Conference Paper

FairSMOE: Mitigating Multi-Attribute Fairness Problem with Sparse Mixture-of-Experts

  • Changdi Yang
  • Zheng Zhan
  • Ci Zhang
  • Yifan Gong
  • Yize Li
  • Zichong Meng
  • Jun Liu
  • Xuan Shen

Real‐world datasets usually contain multiple attributes, making it essential to ensure fairness across all of them simultaneously. However, different attributes may vary in difficulty, and no existing approaches have effectively addressed this issue. Consequently, an attribute‐adaptive strategy is needed to achieve fairness for all attributes. Multi‐task Learning (MTL) leverages shared information to optimize multiple tasks concurrently, while Sparsely‐Gated Mixture‐of‐Experts (SMoE) can dynamically allocate computational resources to the most needed tasks. In this work, we formulate multi‐attribute fairness issue as an MTL problem and employ SMoE to achieve desirable performance across all attributes simultaneously. We first analyze the feasibility and find the potentiality by formalizing multi-attribute fairness problem into a MTL problem and mitigating it by using SMoE. However, vanilla SMoE could lead to over-utilization problem which causes sub-optimal performance. We then proposed an innovative SMoE framework for multi-attribute fair image classification, which further improves multi-attribute fairness by redesigning the MoE layer and routing policy with fairness consideration. Extensive experiments demonstrated the effectiveness. Taking a DeiT-Small as the backbone, we achieve 77. 25% and 86. 01% accuracy on the ISIC2019 and CelebA dataset respectively with Multi-attribute Predictive Quality Disparity (PQD) score of 0. 801 and 0. 787, beating current state-of-the-art methods Muffin, InfoFair and MultiFair.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

In-Context Meta LoRA Generation

  • Yihua Shao
  • Minxi Yan
  • Yang Liu
  • Siyu Chen
  • Wenjie Chen
  • Xinwei Long
  • Ziyang Yan
  • Lei Li

Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1% storage compared with the original LoRA. The code is available at https: //github. com/YihuaJerry/ICM-LoRA.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition

  • Zhicheng Zhang
  • Hao Tang
  • Jinhui Tang

Given the critical role of birds in ecosystems, Fine-Grained Bird Recognition (FGBR) has gained increasing attention, particularly in distinguishing birds within similar subcategories. Although Vision Transformer (ViT)-based methods often outperform Convolutional Neural Network (CNN)-based methods in FGBR, recent studies reveal that the limited receptive field of plain ViT model hinders representational richness and makes them vulnerable to scale variance. Thus, enhancing the multi-scale capabilities of existing ViT-based models to overcome this bottleneck in FGBR is a worthwhile pursuit. In this paper, we propose a novel framework for FGBR, namely Multi-scale Diverse Cues Modeling (MDCM), which explores diverse cues at different scales across various stages of a multi-scale Vision Transformer (MS-ViT) in an ``Activation-Selection-Aggregation'' paradigm. Specifically, we first propose a multi-scale cue activation module to ensure the discriminative cues learned at different stage are mutually different. Subsequently, a multi-scale token selection mechanism is proposed to remove redundant noise and highlight discriminative, scale-specific cues at each stage. Finally, the selected tokens from each stage are independently utilized for bird recognition, and the recognition results from multiple stages are adaptively fused through a multi-scale dynamic aggregation mechanism for final model decisions. Both qualitative and quantitative results demonstrate the effectiveness of our proposed MDCM, which outperforms CNN- and ViT-based models on several widely-used FGBR benchmarks.

PDF Details DOI

IROS Conference 2025 Conference Paper

OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGB

  • Yunzhi Lin
  • Yipu Zhao
  • Fu-Jen Chu
  • Xingyu Chen
  • Weiyao Wang 0001
  • Hao Tang
  • Patricio A. Vela
  • Matt Feiszli

To address the challenge of short-term object pose tracking in dynamic environments with monocular RGB input, we introduce a large-scale synthetic dataset Omni-Pose6D, crafted to mirror the diversity of real-world conditions. We additionally present a benchmarking framework for a comprehensive comparison of pose tracking algorithms. We propose a pipeline featuring an uncertainty-aware keypoint refinement network, employing probabilistic modeling to refine pose estimation. Comparative evaluations demonstrate that our approach achieves performance superior to existing baselines on real datasets, underscoring the effectiveness of our synthetic dataset and refinement technique in enhancing tracking precision in dynamic contexts. Our contributions set a new precedent for the development and assessment of object pose tracking methodologies in complex scenes.

Details

IJCAI Conference 2025 Conference Paper

OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection

  • Yu Liu
  • Hao Tang
  • Haiqi Zhang
  • Jing Qin
  • Zechao Li

Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications. While zero-shot OOD detection, which requires no training on in-distribution (ID) data, has become feasible with the emergence of vision-language models like CLIP, existing methods primarily focus on semantic matching and fail to fully capture distributional discrepancies. To address these limitations, we propose OT-DETECTOR, a novel framework that employs Optimal Transport (OT) to quantify both semantic and distributional discrepancies between test samples and ID labels. Specifically, we introduce cross-modal transport mass and transport cost as semantic-wise and distribution-wise OOD scores, respectively, enabling more robust detection of OOD samples. Additionally, we present a semantic-aware content refinement (SaCR) module, which utilizes semantic cues from ID labels to amplify the distributional discrepancy between ID and hard OOD samples. Extensive experiments on several benchmarks demonstrate that OT-DETECTOR achieves state-of-the-art performance across various OOD detection tasks, particularly in challenging hard-OOD scenarios.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

PoE-World: Compositional World Modeling with Products of Programmatic Experts

  • Top Piriyakulkij
  • Yichao Liang
  • Hao Tang
  • Adrian Weller
  • Marta Kryven
  • Kevin Ellis

Learning how the world works is central to building AI agents that can adapt to complex environments. Traditional world models based on deep-learning demand vast amounts of training data, and do not flexibly update their knowledge from sparse observations. Recent advances in program synthesis using Large Language Models (LLMs) give an alternate approach which learns world models represented as source code, supporting strong generalization from little data. To date, application of program-structured world models remains limited to natural language and grid-world domains. We introduce a novel program synthesis method for effectively modeling complex, non-gridworld domains by representing a world model as an exponentially-weighted product of programmatic experts (PoE-World) synthesized by LLMs. We show that this approach can learn complex, stochastic world models from just a few observations. We evaluate the learned world models by embedding them in a model-based planning agent, demonstrating efficient performance and generalization to unseen levels on Atari's Pong and Montezuma's Revenge.

PDF Details

NeurIPS Conference 2025 Conference Paper

RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness

  • Fanhu Zeng
  • Haiyang Guo
  • Fei Zhu
  • Li Shen
  • Hao Tang

Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relations for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness.

PDF Details

IJCAI Conference 2025 Conference Paper

Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution

  • Zihang Liu
  • Zhenyu Zhang
  • Hao Tang

Diffusion-based image super-resolution (SR) methods have demonstrated remarkable performance. Recent advancements have introduced deterministic sampling processes that reduce inference from 15 iterative steps to a single step, thereby significantly improving the inference speed of existing diffusion models. However, their efficiency remains limited when handling complex semantic regions due to the single-step inference. To address this limitation, we propose SAMSR, a semantic-guided diffusion framework that incorporates semantic segmentation masks into the sampling process. Specifically, we introduce the SAM-Noise Module, which refines Gaussian noise using segmentation masks to preserve spatial and semantic features. Furthermore, we develop a pixel-wise sampling strategy that dynamically adjusts the residual transfer rate and noise strength based on pixel-level semantic weights, prioritizing semantically rich regions during the diffusion process. To enhance model training, we also propose a semantic consistency loss, which aligns pixel-wise semantic weights between predictions and ground truth. Extensive experiments on both real-world and synthetic datasets demonstrate that SAMSR significantly improves perceptual quality and detail recovery, particularly in semantically complex images.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Stable-Hair: Real-World Hair Transfer via Diffusion Model

  • Yuxuan Zhang
  • Qing Zhang
  • Yiren Song
  • Jichao Zhang
  • Hao Tang
  • Jiaming Liu

Current hair transfer methods struggle to handle diverse and intricate hairstyles, limiting their applicability in real-world scenarios. In this paper, we propose a novel diffusion-based hair transfer framework, named Stable-Hair, which robustly transfers a wide range of real-world hairstyles to user-provided faces for virtual hair try-on. To achieve this goal, our Stable-Hair framework is designed as a two-stage pipeline. In the first stage, we train a Bald Converter alongside stable diffusion to remove hair from the user-provided face images, resulting in bald images. In the second stage, we specifically designed a Hair Extractor and a Latent IdentityNet to transfer the target hairstyle with highly detailed and high-fidelity to the bald image. The Hair Extractor is trained to encode reference images with the desired hairstyles, while the Latent IdentityNet ensures consistency in identity and background. To minimize color deviations between source images and transfer results, we introduce a novel Latent ControlNet architecture, which functions as both the Bald Converter and Latent IdentityNet. After training on our curated triplet dataset, our method accurately transfers highly detailed and high-fidelity hairstyles to the source images. Extensive experiments demonstrate that our approach achieves state-of-the-art performance compared to existing hair transfer methods.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment

  • Jun Liu
  • Zhenglun Kong
  • Pu Zhao
  • Changdi Yang
  • Xuan Shen
  • Hao Tang
  • Geng Yuan
  • Wei Niu

Structured pruning for large language models (LLMs) has garnered significant academic interest due to its ability to efficiently compress and accelerate LLMs by eliminating redundant weight groups at a coarse-grained granularity. Current structured pruning methods for LLMs typically depend on a singular granularity for assessing weight importance, resulting in notable performance degradation in downstream tasks. Intriguingly, our empirical investigations reveal that utilizing unstructured pruning, which achieves better performance retention by pruning weights at a finer granularity, \emph{i.e.}, individual weights, yields significantly varied sparse LLM structures when juxtaposed to structured pruning. This suggests that evaluating both holistic and individual assessments for weight importance are essential for LLM pruning. Building on this insight, we introduce the Hybrid-grained Weight Importance Assessment (HyWIA), a novel method that merges fine-grained and coarse-grained evaluations of weight importance for the pruning of LLMs. Leveraging an attention mechanism, HyWIA adaptively determines the optimal blend of granularity in weight importance assessments in an end-to-end pruning manner. Extensive experiments on LLaMA-V1/V2, Vicuna, Baichuan, and Bloom across various benchmarks demonstrate the effectiveness of HyWIA in pruning LLMs. For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2.82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis

  • Jing Hao
  • Yuxuan Fan
  • Yanpeng Sun
  • Kaixin Guo
  • Lin Lizhuo
  • Jinrong Yang
  • Qiyong Ai
  • Lun Wong

Recent advances in large vision-language models (LVLMs) have demonstrated strong performance on general-purpose medical tasks. However, their effectiveness in specialized domains such as dentistry remains underexplored. In particular, panoramic X-rays, a widely used imaging modality in oral radiology, pose interpretative challenges due to dense anatomical structures and subtle pathological cues, which are not captured by existing medical benchmarks or instruction datasets. To this end, we introduce MMOral, the first large-scale multimodal instruction dataset and benchmark tailored for panoramic X-ray interpretation. MMOral consists of 20, 563 annotated images paired with 1. 3 million instruction-following instances across diverse task types, including attribute extraction, report generation, visual question answering, and image-grounded dialogue. In addition, we present MMOral-Bench, a comprehensive evaluation suite covering five key diagnostic dimensions in dentistry. We evaluate 64 LVLMs on MMOral-Bench and find that even the best-performing model, i. e. , GPT-4o, only achieves 43. 31% accuracy, revealing significant limitations of current models in this domain. To promote the progress of this specific domain, we provide the supervised fine-tuning (SFT) process utilizing our meticulously curated MMOral instruction dataset. Remarkably, a single epoch of SFT yields substantial performance enhancements for LVLMs, e. g. , Qwen2. 5-VL-7B demonstrates a 24. 73% improvement. MMOral holds significant potential as a critical foundation for intelligent dentistry and enables more clinically impactful multimodal AI systems in the dental field.

PDF Details

IROS Conference 2025 Conference Paper

TTTFusion: A Test-Time Training-Based Strategy for Multimodal Medical Image Fusion in Surgical Robots

  • Qinhua Xie
  • Hao Tang

With the increasing use of surgical robots in clinical practice, enhancing their ability to process multimodal medical images has become a key research challenge. Although traditional medical image fusion methods have made progress in improving fusion accuracy, they still face significant challenges in real-time performance, fine-grained feature extraction, and edge preservation. In this paper, we introduce TTTFusion, a Test-Time Training (TTT)-based image fusion strategy that dynamically adjusts model parameters during inference to efficiently fuse multimodal medical images. By adapting the model during the test phase, our method optimizes the parameters based on the input image data, leading to improved accuracy and better detail preservation in the fusion results. Experimental results demonstrate that TTTFusion significantly enhances the fusion quality of multimodal images compared to traditional fusion methods, particularly in fine-grained feature extraction and edge preservation. This approach not only improves image fusion accuracy but also offers a novel technical solution for real-time image processing in surgical robots.

Details

EAAI Journal 2025 Journal Article

Two-stream structure-oriented neighbor enhancement network for dental model segmentation

  • Zhihua Liu
  • Hao Tang
  • Jiutao Xue
  • Yuhe Liao

The primary objective of digital orthodontic treatment is to achieve accurate tooth segmentation of the three-dimensional (3D) dental mesh model obtained from oral scanning equipment. However, current advanced deep learning-based methods often consolidate all features into a single vector during the feature learning process, thus overlooking the distinct information among features and subsequently weakening their complementary roles. To address this issue, we propose a two-stream structure-oriented neighbor enhancement network (TSNEN) to improve the complementary effect among different features. Specifically, TSNEN develops an input-specific two-stream structure and feature enhancement modules to emphasize the geometric disparities among various meshes and achieve the complementary enhancement of different features, respectively. Furthermore, the self-attention module is modified to fully integrate the local features derived from the branches of the two streams, which effectively balances the data differences among different features to mitigate feature confusion, and ultimately achieves accurate segmentation of the dental model. The real dental model dataset is analyzed to verify the effectiveness and capability of the proposed method which reached an overall accuracy (OA) at 96. 83 % and mean over union (mIoU) at 92. 06 %. Finally, the comparative analysis is implemented and the results further show that the proposed method has better performance both in prediction accuracy and robustness.

Details DOI

NeurIPS Conference 2025 Conference Paper

UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface

  • Hao Tang
  • Chen-Wei Xie
  • Haiyang Wang
  • Xiaoyi Bao
  • Tingyu Weng
  • Pandeng Li
  • Yun Zheng
  • Liwei Wang

Generalist models have achieved remarkable success in both language and vision-language tasks, showcasing the potential of unified modeling. However, effectively integrating fine-grained perception tasks like detection and segmentation into these models remains a significant challenge. This is primarily because these tasks often rely heavily on task-specific designs and architectures that can complicate the modeling process. To address this challenge, we present UFO, a framework that unifies fine-grained visual perception tasks through an open-ended language interface. By transforming all perception targets into the language space, UFO unifies object-level detection, pixel-level segmentation, and image-level vision-language tasks into a single model. Additionally, we introduce a novel embedding retrieval approach that relies solely on the language interface to support segmentation tasks. Our framework bridges the gap between fine-grained perception and vision-language tasks, significantly simplifying architectural design and training strategies while achieving comparable or superior performance to methods with intricate task-specific designs. After multi-task training on five standard visual perception datasets, UFO outperforms the previous state-of-the-art generalist models by 12. 3 mAP on COCO instance segmentation and 3. 3 mIoU on ADE20K semantic segmentation. Furthermore, our method seamlessly integrates with existing MLLMs, effectively combining fine-grained perception capabilities with their advanced language abilities, thereby achieving superior performance on the challenging reasoning segmentation. Code and models are available at https: //github. com/nnnth/UFO.

PDF Details

NeurIPS Conference 2024 Conference Paper

Code Repair with LLMs gives an Exploration-Exploitation Tradeoff

  • Hao Tang
  • Keya Hu
  • Jin P. Zhou
  • Sicheng Zhong
  • Wei-Long Zheng
  • Xujie Si
  • Kevin Ellis

Iteratively improving and repairing source code with large language models (LLMs), known as refinement, has emerged as a popular way of generating programs that would be too complex to construct in one shot. Given a bank of test cases, together with a candidate program, an LLM can improve that program by being prompted with failed test cases. But it remains an open question how to best iteratively refine code, with prior work employing simple greedy or breadth-first strategies. We show here that refinement exposes an explore-exploit tradeoff: exploit by refining the program that passes the most test cases, or explore by refining a lesser considered program. We frame this as an arm-acquiring bandit problem, which we solve with Thompson Sampling. The resulting LLM-based program synthesis algorithm is broadly applicable: Across loop invariant synthesis, visual reasoning puzzles, and competition programming problems, we find that our new method can solve more problems using fewer language model calls.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Delving into Multimodal Prompting for Fine-Grained Visual Classification

  • Xin Jiang
  • Hao Tang
  • Junyao Gao
  • Xiaoyu Du
  • Shengfeng He
  • Zechao Li

Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

PDF Details DOI

TIST Journal 2024 Journal Article

Enabling Graph Neural Networks for Semi-Supervised Risk Prediction in Online Credit Loan Services

  • Hao Tang
  • Cheng Wang
  • Jianguo Zheng
  • Changjun Jiang

Graph neural networks (GNNs) are playing exciting roles in the application scenarios where features are hidden in information associations. Fraud prediction of online credit loan services (OCLSs) is such a typical scenario. But it has another rather critical challenge, i.e., the scarcity of data labels. Fortunately, GNNs can also cope with this problem due to their good ability of semi-supervised learning by mining structure and feature information within graphs. Nevertheless, the gain of internal information is often too limited to help GNNs handle the extreme deficiency of labels with high performance beyond the basic requirement of fraud prediction in OCLSs. Therefore, adding labels from the experts, such as manually adding labels through rules, has become a logical practice. However, the existing rule engines for OCLSs have the confliction problem among continuously accumulated rules. To address this issue, we propose a Snorkel-based Semi-Supervised GNN (S3GNN). Under S3GNN, we specially design an upgraded version of the rule engines, called Graph-Oriented Snorkel (GOS), a graph-specific extension of Snorkel, a widely used weakly supervised learning framework, to design rules by subject matter experts (SMEs) and resolve confliction. In particular, in the graph of an anti-fraud scenario, each node pair may have multiple different types of edges, so we propose the Multiple Edge-Types Based Attention mechanism. In general, for the heterogeneous information and multiple relations in the graph, we first obtain the embedding of applicant nodes by aggregating the representation of attribute nodes, and then use the attention mechanism to aggregate neighbor nodes on multiple meta-paths to get ultimate applicant node embedding. We conduct experiments over the real-life data of a large financial platform. The results demonstrate that S3GNN can outperform the state-of-the-art methods, including the method of pilot platform.

Details DOI

AAAI Conference 2024 Conference Paper

G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model

  • Pan Xie
  • Qipeng Zhang
  • Peng Taiying
  • Hao Tang
  • Yao Du
  • Zexian Li

The Sign Language Production (SLP) project aims to automatically translate spoken languages into sign sequences. Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose sequences (G2P). In this paper, we present a novel solution for this task by converting the continuous pose space generation problem into a discrete sequence generation problem. We introduce the Pose-VQVAE framework, which combines Variational Autoencoders (VAEs) with vector quantization to produce a discrete latent representation for continuous pose sequences. Additionally, we propose the G2P-DDM model, a discrete denoising diffusion architecture for length-varied discrete sequence data, to model the latent prior. To further enhance the quality of pose sequence generation in the discrete space, we present the CodeUnet model to leverage spatial-temporal information. Lastly, we develop a heuristic sequential clustering method to predict variable lengths of pose sequences for corresponding gloss sequences. Our results show that our model outperforms state-of-the-art G2P models on the public SLP evaluation benchmark. For more generated results, please visit our project page: https://slpdiffusier.github.io/g2p-ddm.

PDF Details DOI

AIIM Journal 2024 Journal Article

Integrated block-wise neural network with auto-learning search framework for finger gesture recognition using sEMG signals

  • Shurun Wang
  • Hao Tang
  • Feng Chen
  • Qi Tan
  • Qi Jiang

Accurate finger gesture recognition with surface electromyography (sEMG) is essential and long-challenge in the muscle-computer interface, and many high-performance deep learning models have been developed to predict gestures. For these models, problem-specific tuning of network architecture is essential for improving the performance, yet it requires substantial knowledge of network architecture design and commitment of time and effort. This process thus imposes a major obstacle to the widespread and flexible application of modern deep learning. To address this issue, we present an auto-learning search framework (ALSF) to generate the integrated block-wised neural network (IBWNN) for sEMG-based gesture recognition. IBWNN contains several feature extraction blocks and dimensional reduction layers, and each feature extraction block integrates two sub-blocks (i. e. , multi-branch convolutional block and triplet attention block). Meanwhile, ALSF generates optimal models for gesture recognition through the reinforcement learning method. The results show that the generated models yield state-of-the-art results compared to the modern popular networks on the open dataset Ninapro DB5. Moreover, compared to other networks, the generated models have fewer parameters and can be deployed in practical applications with less resource consumption.

Details DOI

JBHI Journal 2024 Journal Article

Multi-Domain Based Dynamic Graph Representation Learning for EEG Emotion Recognition

  • Hao Tang
  • Songyun Xie
  • Xinzhou Xie
  • Yujie Cui
  • Bohan Li
  • Dalu Zheng
  • Yu Hao
  • Xiangming Wang

Graph neural networks (GNNs) have demonstrated efficient processing of graph-structured data, making them a promising method for electroencephalogram (EEG) emotion recognition. However, due to dynamic functional connectivity and nonlinear relationships between brain regions, representing EEG as graph data remains a great challenge. To solve this problem, we proposed a multi-domain based graph representation learning (MD $^{2}$ GRL) framework to model EEG signals as graph data. Specifically, MD $^{2}$ GRL leverages gated recurrent units (GRU) and power spectral density (PSD) to construct node features of two subgraphs. Subsequently, the self-attention mechanism is adopted to learn the similarity matrix between nodes and fuse it with the intrinsic spatial matrix of EEG to compute the corresponding adjacency matrix. In addition, we introduced a learnable soft thresholding operator to sparsify the adjacency matrix to reduce noise in the graph structure. In the downstream task, we designed a dual-branch GNN and incorporated spatial asymmetry for graph coarsening. We conducted experiments using the publicly available datasets SEED and DEAP, separately for subject-dependent and subject-independent, to evaluate the performance of our model in emotion classification. Experimental results demonstrated that our method achieved state-of-the-art (SOTA) classification performance in both subject-dependent and subject-independent experiments. Furthermore, the visualization analysis of the learned graph structure reveals EEG channel connections that are significantly related to emotion and suppress irrelevant noise. These findings are consistent with established neuroscience research and demonstrate the potential of our approach in comprehending the neural underpinnings of emotion.

Details DOI

NeurIPS Conference 2024 Conference Paper

Revisiting Adversarial Patches for Designing Camera-Agnostic Attacks against Person Detection

  • Hui Wei
  • Zhixiang Wang
  • Kewei Zhang
  • Jiaqi Hou
  • Yuanwei Liu
  • Hao Tang
  • Zheng Wang

Physical adversarial attacks can deceive deep neural networks (DNNs), leading to erroneous predictions in real-world scenarios. To uncover potential security risks, attacking the safety-critical task of person detection has garnered significant attention. However, we observe that existing attack methods overlook the pivotal role of the camera, involving capturing real-world scenes and converting them into digital images, in the physical adversarial attack workflow. This oversight leads to instability and challenges in reproducing these attacks. In this work, we revisit patch-based attacks against person detectors and introduce a camera-agnostic physical adversarial attack to mitigate this limitation. Specifically, we construct a differentiable camera Image Signal Processing (ISP) proxy network to compensate for the physical-to-digital transition gap. Furthermore, the camera ISP proxy network serves as a defense module, forming an adversarial optimization framework with the attack module. The attack module optimizes adversarial patches to maximize effectiveness, while the defense module optimizes the conditional parameters of the camera ISP proxy network to minimize attack effectiveness. These modules engage in an adversarial game, enhancing cross-camera stability. Experimental results demonstrate that our proposed Camera-Agnostic Patch (CAP) attack effectively conceals persons from detectors across various imaging hardware, including two distinct cameras and four smartphones.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

  • Hao Tang
  • Darren Key
  • Kevin Ellis

We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment. The world model tries to explain its interactions, while also being optimistic about what reward it can achieve. We define this optimism as a logical constraint between a program and a planner. We study our agent on gridworlds, and on task planning, finding our approach is more sample-efficient compared to deep RL, more compute-efficient compared to ReAct-style agents, and that it can transfer its knowledge across environments by editing its code.

PDF Details DOI

TMLR Journal 2023 Journal Article

3D-Aware Video Generation

  • Sherwin Bahmani
  • Jeong Joon Park
  • Despoina Paschalidou
  • Hao Tang
  • Gordon Wetzstein
  • Leonidas Guibas
  • Luc Van Gool
  • Radu Timofte

Generative models have emerged as an essential building block for many image synthesis and editing tasks. Recent advances in this field have also enabled high-quality 3D or video content to be generated that exhibits either multi-view or temporal consistency. With our work, we explore 4D generative adversarial networks (GANs) that learn unconditional generation of 3D-aware videos. By combining neural implicit representations with time-aware discriminator, we develop a GAN framework that synthesizes 3D video supervised only with monocular videos. We show that our method learns a rich embedding of decomposable 3D structures and motions that enables new visual effects of spatio-temporal renderings while producing imagery with quality comparable to that of existing 3D or video GANs.

PDF Details

IJCAI Conference 2023 Conference Paper

Data Level Lottery Ticket Hypothesis for Vision Transformers

  • Xuan Shen
  • Zhenglun Kong
  • Minghai Qin
  • Peiyan Dong
  • Geng Yuan
  • Xin Meng
  • Hao Tang
  • Xiaolong Ma

The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method, called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input data consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the winning tickets, which represent a significant amount of information in the input data. We use a ticket selector to generate the winning tickets based on the informativeness of patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. The experiments show that there is a clear difference between the performance of models trained with winning tickets and randomly selected subsets, which verifies our proposed theory. We elaborate the analogical similarity between our proposed Data-LTH-ViTs and the conventional LTH for further verifying the integrity of our theory. The Source codes are available at https: //github. com/shawnricecake/vit-lottery-ticket-input.

PDF Details DOI

AAAI Conference 2023 Conference Paper

DE-net: Dynamic Text-Guided Image Editing Adversarial Networks

  • Ming Tao
  • Bing-Kun Bao
  • Hao Tang
  • Fei Wu
  • Longhui Wei
  • Qi Tian

Text-guided image editing models have shown remarkable results. However, there remain two problems. First, they employ fixed manipulation modules for various editing requirements (e.g., color changing, texture changing, content adding and removing), which results in over-editing or insufficient editing. Second, they do not clearly distinguish between text-required and text-irrelevant parts, which leads to inaccurate editing. To solve these limitations, we propose: (i) a Dynamic Editing Block (DEBlock) that composes different editing modules dynamically for various editing requirements. (ii) a Composition Predictor (Comp-Pred), which predicts the composition weights for DEBlock according to the inference on target texts and source images. (iii) a Dynamic text-adaptive Convolution Block (DCBlock) that queries source image features to distinguish text-required parts and text-irrelevant parts. Extensive experiments demonstrate that our DE-Net achieves excellent performance and manipulates source images more correctly and accurately.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Does Graph Distillation See Like Vision Dataset Counterpart?

  • Beining Yang
  • Kai Wang
  • Qingyun Sun
  • Cheng Ji
  • Xingcheng Fu
  • Hao Tang
  • Yang You
  • Jianxin Li

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (\textbf{SGDD}) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD~across 9 datasets and achieve state-of-the-art results on all of them: for example, on YelpChi dataset, our approach maintains 98. 6\% test accuracy of training on the original graph dataset with 1, 000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17. 6\% $\sim$ 31. 4\% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code will be made public.

PDF Details

NeurIPS Conference 2023 Conference Paper

EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset

  • Hao Tang
  • Kevin J Liang
  • Kristen Grauman
  • Matt Feiszli
  • Weiyao Wang

Visual object tracking is a key component to many egocentric vision problems. However, the full spectrum of challenges of egocentric tracking faced by an embodied AI is underrepresented in many existing datasets; these tend to focus on relatively short, third-person videos. Egocentric video has several distinguishing characteristics from those commonly found in past datasets: frequent large camera motions and hand interactions with objects commonly lead to occlusions or objects exiting the frame, and object appearance can change rapidly due to widely different points of view, scale, or object states. Embodied tracking is also naturally long-term, and being able to consistently (re-)associate objects to their appearances and disappearances over as long as a lifetime is critical. Previous datasets under-emphasize this re-detection problem, and their "framed" nature has led to adoption of various spatiotemporal priors that we find do not necessarily generalize to egocentric video. We thus introduce EgoTracks, a new dataset for long-term egocentric visual object tracking. Sourced from the Ego4D dataset, this new dataset presents a significant challenge to recent state-of-the-art single-object tracking models, which we find score poorly on traditional tracking metrics for our new dataset, compared to popular benchmarks. We further show improvements that can be made to a STARK tracker to significantly increase its performance on egocentric data, resulting in a baseline model we call EgoSTARK. We publicly release our annotations and benchmark, hoping our dataset leads to further advancements in tracking.

PDF Details

NeurIPS Conference 2023 Conference Paper

HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception

  • Peiyan Dong
  • Zhenglun Kong
  • Xin Meng
  • Pinrui Yu
  • Yifan Gong
  • Geng Yuan
  • Hao Tang
  • Yanzhi Wang

The bird's-eye-view (BEV) perception plays a critical role in autonomous driving systems, involving the accurate and efficient detection and tracking of objects from a top-down perspective. To achieve real-time decision-making in self-driving scenarios, low-latency computation is essential. While recent approaches to BEV detection have focused on improving detection precision using Lift-Splat-Shoot (LSS)-based or transformer-based schemas, the substantial computational and memory burden of these approaches increases the risk of system crashes when multiple on-vehicle tasks run simultaneously. Unfortunately, there is a dearth of literature on efficient BEV detector paradigms, let alone achieving realistic speedups. Unlike existing works that focus on reducing computation costs, this paper focuses on developing an efficient model design that prioritizes actual on-device latency. To achieve this goal, we propose a latency-aware design methodology that considers key hardware properties, such as memory access cost and degree of parallelism. Given the prevalence of GPUs as the main computation platform for autonomous driving systems, we develop a theoretical latency prediction model and introduce efficient building operators. By leveraging these operators and following an effective local-to-global visual modeling process, we propose a hardware-oriented backbone that is also optimized for strong feature capturing and fusing. Using these insights, we present a new hardware-oriented framework for efficient yet accurate camera-view BEV detectors. Experiments show that HotBEV achieves a 2\%$\sim$23\% NDS gain, and 2\%$\sim$7. 8\% mAP gain with a 1. 1$\times$$\sim$3. 4$\times$ speedups compared to existing works on V100; On multiple GPU devices such as GPU GTX 2080 and the low-end GTX 1080, HotBEV achieves 1. 1$\times$$\sim$6. 3$\times$ faster than others.

PDF Details

AAAI Conference 2023 Conference Paper

HOTCOLD Block: Fooling Thermal Infrared Detectors with a Novel Wearable Design

  • Hui Wei
  • Zhixiang Wang
  • Xuemei Jia
  • Yinqiang Zheng
  • Hao Tang
  • Shin'ichi Satoh
  • Zheng Wang

Adversarial attacks on thermal infrared imaging expose the risk of related applications. Estimating the security of these systems is essential for safely deploying them in the real world. In many cases, realizing the attacks in the physical space requires elaborate special perturbations. These solutions are often impractical and attention-grabbing. To address the need for a physically practical and stealthy adversarial attack, we introduce HotCold Block, a novel physical attack for infrared detectors that hide persons utilizing the wearable Warming Paste and Cooling Paste. By attaching these readily available temperature-controlled materials to the body, HotCold Block evades human eyes efficiently. Moreover, unlike existing methods that build adversarial patches with complex texture and structure features, HotCold Block utilizes an SSP-oriented adversarial optimization algorithm that enables attacks with pure color blocks and explores the influence of size, shape, and position on attack performance. Extensive experimental results in both digital and physical environments demonstrate the performance of our proposed HotCold Block. Code is available: https://github.com/weihui1308/HOTCOLDBlock.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

LART: Neural Correspondence Learning with Latent Regularization Transformer for 3D Motion Transfer

  • Haoyu Chen
  • Hao Tang
  • Radu Timofte
  • Luc V Gool
  • Guoying Zhao

3D motion transfer aims at transferring the motion from a dynamic input sequence to a static 3D object and outputs an identical motion of the target with high-fidelity and realistic visual effects. In this work, we propose a novel 3D Transformer framework called LART for 3D motion transfer. With carefully-designed architectures, LART is able to implicitly learn the correspondence via a flexible geometry perception. Thus, unlike other existing methods, LART does not require any key point annotations or pre-defined correspondence between the motion source and target meshes and can also handle large-size full-detailed unseen 3D targets. Besides, we introduce a novel latent metric regularization on the Transformer for better motion generation. Our rationale lies in the observation that the decoded motions can be approximately expressed as linearly geometric distortion at the frame level. The metric preservation of motions could be translated to the formation of linear paths in the underlying latent space as a rigorous constraint to control the synthetic motions occurring in the construction of the latent space. The proposed LART shows a high learning efficiency with the need for a few samples from the AMASS dataset to generate motions with plausible visual effects. The experimental results verify the potential of our generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The code is made available: https: //github. com/mikecheninoulu/LART.

PDF Details

ICLR Conference 2023 Conference Paper

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

  • Chao Yu 0005
  • Jiaxuan Gao
  • Weilin Liu
  • Botian Xu
  • Hao Tang
  • Jiaqi Yang
  • Yu Wang 0002
  • Yi Wu 0013

There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.

Details

NeurIPS Conference 2023 Conference Paper

Object Reprojection Error (ORE): Camera pose benchmarks from lightweight tracking annotations

  • Xingyu Chen
  • Weiyao Wang
  • Hao Tang
  • Matt Feiszli

3D spatial understanding is highly valuable in the context of semantic modeling of environments, agents, and their relationships. Semantic modeling approaches employed on monocular video often ingest outputs from off-the-shelf SLAM/SfM pipelines, which are anecdotally observed to perform poorly or fail completely on some fraction of the videos of interest. These target videos may vary widely in complexity of scenes, activities, camera trajectory, etc. Unfortunately, such semantically-rich video data often comes with no ground-truth 3D information, and in practice it is prohibitively costly or impossible to obtain ground truth reconstructions or camera pose post-hoc. This paper proposes a novel evaluation protocol, Object Reprojection Error (ORE) to benchmark camera trajectories; ORE computes reprojection error for static objects within the video and requires only lightweight object tracklet annotations. These annotations are easy to gather on new or existing video, enabling ORE to be calculated on essentially arbitrary datasets. We show that ORE maintains high rank correlation with standard metrics based on groundtruth. Leveraging ORE, we source videos and annotations from Ego4D-EgoTracks, resulting in EgoStatic, a large-scale diverse dataset for evaluating camera trajectories in-the-wild.

PDF Details

NeurIPS Conference 2023 Conference Paper

PackQViT: Faster Sub-8-bit Vision Transformers via Full and Packed Quantization on the Mobile

  • Peiyan Dong
  • Lei Lu
  • Chao Wu
  • Cheng Lyu
  • Geng Yuan
  • Hao Tang
  • Yanzhi Wang

While Vision Transformers (ViTs) have undoubtedly made impressive strides in computer vision (CV), their intricate network structures necessitate substantial computation and memory resources. A decision-making process for CV tasks typically entails performing computations with low latency, which is a tricky problem for ViT models. Model quantization is a widely-used technique to optimize the hardware efficiency of deep neural networks. Full quantization under Sub-8-bit precision, in particular, is a promising solution to reduce inference latency significantly. Unfortunately, current commodity hardware, such as CPUs and GPUs, still struggles to efficiently execute these sub-8-bit quantized networks, as their SIMD instructions only support a granularity of 8 bits or wider. Also, there is a scarcity of literature that presents a full quantization paradigm for ViTs. In this paper, we propose an activation-aware fully sub-8-bit quantization-aware training (QAT) framework called PackQViT for efficient yet accurate ViT acceleration on mobile devices to facilitate real-time AI-powered decision-making. Specifically, in revisiting data activation within the ViT dataflow, two characteristics are relevant to quantization strategy and precision: the long-tailed distribution and systematic channel-wise outliers. In response, we employ either log2 quantization or clipping to address the long-tailed distribution and incorporate outlier-aware training for residual link quantization to regulate the various channel-wise outliers more consistently. Notably, due to the systematic fixed pattern, outlier-aware training approach can predict the channel indices and regularized scales of outliers in advance, thus avoiding the runtime data-adaptive selection during inference. Furthermore, we employ Int-$2^{n}$-Softmax, Int-LayerNorm, and Integer GELU to enable integer-only computation flow. Finally, we develop a SIMD-based 4-bit packed multiplier to achieve end-to-end ViT acceleration on mobile phones. Compared to prior studies on ViT quantization using 8-bit precision, PackQViT surpasses other works by an improved accuracy ranging from 0. 4\% to 17. 9\% for various widely used ViTs on ImageNet dataset; under 4-bit precision, PackQViT demonstrates 0. 4%$\sim$2. 8% higher accuracy. Compared to the baseline multiplier, our implementations on the Realme GT Android smartphone with Snapdragon 870 SoC CPU achieve 2. 6x$\sim$3. 7x speedup under 8-bit scenario and 3. 8x$\sim$5. 9x speedup under 4-bit which ensures practical real-time performance.

PDF Details

AAAI Conference 2023 Conference Paper

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

  • Zhenglun Kong
  • Haoyu Ma
  • Geng Yuan
  • Mengshu Sun
  • Yanyue Xie
  • Peiyan Dong
  • Xin Meng
  • Xuan Shen

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT. Our code is released at https://github.com/ZLKong/Tri-Level-ViT

PDF Details DOI

IJCAI Conference 2023 Conference Paper

RZCR: Zero-shot Character Recognition via Radical-based Reasoning

  • Xiaolei Diao
  • Daqian Shi
  • Hao Tang
  • Qiang Shen
  • Yanzeng Li
  • Lei Wu
  • Hao Xu

The long-tail effect is a common issue that limits the performance of deep learning models on real-world datasets. Character image datasets are also affected by such unbalanced data distribution due to differences in character usage frequency. Thus, current character recognition methods are limited when applied in the real world, especially for the categories in the tail that lack training samples, e. g. , uncommon characters. In this paper, we propose a zero-shot character recognition framework via radical-based reasoning, called RZCR, to improve the recognition performance of few-sample character categories in the tail. Specifically, we exploit radicals, the graphical units of characters, by decomposing and reconstructing characters according to orthography. RZCR consists of a visual semantic fusion-based radical information extractor (RIE) and a knowledge graph character reasoner (KGR). RIE aims to recognize candidate radicals and their possible structural relations from character images in parallel. The results are then fed into KGR to recognize the target character by reasoning with a knowledge graph. We validate our method on multiple datasets, and RZCR shows promising experimental results, especially on few-sample character datasets.

PDF Details DOI

YNICL Journal 2023 Journal Article

Spontaneous beta power, motor-related beta power and cortical thickness in major depressive disorder with psychomotor disturbance

  • Yi Xia
  • Hao Sun
  • Lingling Hua
  • Zhongpeng Dai
  • Xiaoqin Wang
  • Hao Tang
  • Yinglin Han
  • Yishan Du

INTRODUCTION: The psychomotor disturbance is a common symptom in patients with major depressive disorder (MDD). The neurological mechanisms of psychomotor disturbance are intricate, involving alterations in the structure and function of motor-related regions. However, the relationship among changes in the spontaneous activity, motor-related activity, local cortical thickness, and psychomotor function remains unclear. METHOD: A total of 140 patients with MDD and 68 healthy controls performed a simple right-hand visuomotor task during magnetoencephalography (MEG) scanning. All patients were divided into two groups according to the presence of psychomotor slowing. Spontaneous beta power, movement-related beta desynchronization (MRBD), absolute beta power during movement and cortical characteristics in the bilateral primary motor cortex were compared using general linear models with the group as a fixed effect and age as a covariate. Finally, the moderated mediation model was tested to examine the relationship between brain metrics with group differences and psychomotor performance. RESULTS: The patients with psychomotor slowing showed higher spontaneous beta power, movement-related beta desynchronization and absolute beta power during movement than patients without psychomotor slowing. Compared with the other two groups, significant decreases were found in cortical thickness of the left primary motor cortex in patients with psychomotor slowing. Our moderated mediation model showed that the increased spontaneous beta power indirectly affected impaired psychomotor performance by abnormal MRBD, and the indirect effects were moderated by cortical thickness. CONCLUSION: These results suggest that patients with MDD have aberrant cortical beta activity at rest and during movement, combined with abnormal cortical thickness, contributing to the psychomotor disturbance observed in this patient population.

Details DOI

YNICL Journal 2023 Journal Article

The relationship between disrupted anhedonia-related circuitry and suicidal ideation in major depressive disorder: A network-based analysis

  • Xiaoqin Wang
  • Yi Xia
  • Rui Yan
  • Huan Wang
  • Hao Sun
  • Yinghong Huang
  • Lingling Hua
  • Hao Tang

BACKGROUND: Several epidemiological studies and psychological models have suggested that major depressive disorder (MDD) with anhedonia is associated with suicidal ideation (SI). However, little is known about whether the functional network pattern and intrinsic topologically disrupted in patients with anhedonia are related to SI. METHODS: The resting-fMRI by applying network-based statistic (NBS) and graph-theory analyses was estimated in 273 patients with MDD (144 high anhedonia [HA], 129 low anhedonia [LA]) and 150 healthy controls. In addition, we quantified the SI scores of each patient. Finally, the mediation analysis assessed whether anhedonia symptoms could mediate the relationship between anhedonia-related network metrics and SI. RESULT: The NBS analysis demonstrated that individuals with HA have a single abnormally increased functional connectivity component in a frontal-limbic circuit (termed the "anhedonia-related network", including the frontal cortex, striatum, anterior cingulate cortex and amygdala). The graph-theory analysis demonstrated that the anhedonia-related network showed a significantly disrupted topological organization (lower gamma and lambda), which the small-world property trend randomized. Furthermore, the anhedonia symptoms could mediate the relationship between the anhedonia-related network metrics (the mean functional connectivity values, the area under the curves values of gamma and nodal local efficiency in nucleus accumbens) and SI. CONCLUSIONS: We found that disruption of the reward-related network in MDD leads to SI through anhedonia symptoms. These findings show the abnormal topological construction of functional brain network organization in anhedonia, shedding light on the neurological processes underlying SI in MDD patients with anhedonia symptoms.

Details DOI

AAAI Conference 2023 Conference Paper

Towards Real-Time Segmentation on the Edge

  • Yanyu Li
  • Changdi Yang
  • Pu Zhao
  • Geng Yuan
  • Wei Niu
  • Jiexiong Guan
  • Hao Tang
  • Minghai Qin

The research in real-time segmentation mainly focuses on desktop GPUs. However, autonomous driving and many other applications rely on real-time segmentation on the edge, and current arts are far from the goal. In addition, recent advances in vision transformers also inspire us to re-design the network architecture for dense prediction task. In this work, we propose to combine the self attention block with lightweight convolutions to form new building blocks, and employ latency constraints to search an efficient sub-network. We train an MLP latency model based on generated architecture configurations and their latency measured on mobile devices, so that we can predict the latency of subnets during search phase. To the best of our knowledge, we are the first to achieve over 74% mIoU on Cityscapes with semi-real-time inference (over 15 FPS) on mobile GPU from an off-the-shelf phone.

PDF Details DOI

JBHI Journal 2022 Journal Article

Continuous Estimation of Human Joint Angles From sEMG Using a Multi-Feature Temporal Convolutional Attention-Based Network

  • Shurun Wang
  • Hao Tang
  • Lifu Gao
  • Qi Tan

Intention recognition based on surface electromyography (sEMG) signals is pivotal in human-machine interaction (HMI), where continuous motion estimation with high accuracy has been the challenge. The convolutional neural network (CNN) possesses excellent feature extraction capability. Still, it is difficult for ordinary CNN to explore the dependencies of time-series data, so most researchers adopt the recurrent neural network or its variants (e. g. , LSTM) for motion estimation tasks. This paper proposes a multi-feature temporal convolutional attention-based network (MFTCAN) to recognize joint angles continuously. First, we recruited ten subjects to accomplish the signal acquisition experiments in different motion patterns. Then, we developed a joint training mechanism that integrates MFTCAN with commonly used statistical algorithms, and the integrated architectures were named MFTCAN-KNR, MFTCAN-SVR and MFTCAN-LR. Last, we utilized two performance indicators (RMSE and $\text{R}^{2}$ ) to evaluate the effect of different methods. Moreover, we further validated the performance of the proposed method on the open dataset (Ninapro DB2). When evaluating on the original dataset, the average RMSE of the estimations obtained by MFTCAN-KNR is 0. 14, which is significantly less than the results obtained by LSTM (0. 20) and BP (0. 21). The average $\text{R}^{2}$ of the estimations obtained by MFTCAN-KNR is 0. 87, indicating the anti-disturbance ability of the architecture. Moreover, MFTCAN-KNR also achieves high performance when evaluating on the open dataset. The proposed methods can effectively accomplish the task of motion estimation, allowing further implementations in the human-exoskeleton interaction systems.

Details DOI

AAAI Conference 2022 Conference Paper

Geometry-Contrastive Transformer for Generalized 3D Pose Transfer

  • Haoyu Chen
  • Hao Tang
  • Zitong Yu
  • Nicu Sebe
  • Guoying Zhao

We present a customized 3D mesh Transformer model for the pose transfer task. As the 3D pose transfer essentially is a deformation procedure dependent on the given meshes, the intuition of this work is to perceive the geometric inconsistency between the given meshes with the powerful self-attention mechanism. Specifically, we propose a novel geometry-contrastive Transformer that has an efficient 3D structured perceiving ability to the global geometric inconsistencies across the given meshes. Moreover, locally, a simple yet efficient central geodesic contrastive loss is further proposed to improve the regional geometric-inconsistency learning. At last, we present a latent isometric regularization module together with a novel semi-synthesized dataset for the cross-dataset 3D pose transfer task towards unknown spaces. The massive experimental results prove the efficacy of our approach by showing state-of-the-art quantitative performances on SMPL-NPT, FAUST and our new proposed dataset SMG- 3D datasets, as well as promising qualitative results on MGcloth and SMAL datasets. It’s demonstrated that our method can achieve robust 3D pose transfer and be generalized to challenging meshes from unknown spaces on cross-dataset tasks. The code and dataset are made available. Code is available: https: //github. com/mikecheninoulu/CGT.

PDF Details

AAAI Conference 2022 Conference Paper

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

  • Yidi Li
  • Hong Liu
  • Hao Tang

Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98. 6% and 78. 3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

PDF Details

IJCAI Conference 2022 Conference Paper

Real-Time Portrait Stylization on the Edge

  • Yanyu Li
  • Xuan Shen
  • Geng Yuan
  • Jiexiong Guan
  • Wei Niu
  • Hao Tang
  • Bin Ren
  • Yanzhi Wang

In this work we demonstrate real-time portrait stylization, specifically, translating self-portrait into cartoon or anime style on mobile devices. We propose a latency-driven differentiable architecture search method, maintaining realistic generative quality. With our framework, we obtain 10× computation reduction on the generative model and achieve real-time video stylization on off-the-shelf smartphone using mobile GPUs.

PDF Details DOI

NeurIPS Conference 2020 Conference Paper

Belief Propagation Neural Networks

  • Jonathan Kuck
  • Shuvam Chakraborty
  • Hao Tang
  • Rachel Luo
  • Jiaming Song
  • Ashish Sabharwal
  • Stefano Ermon

Learned neural solvers have successfully been used to solve combinatorial optimization and decision problems. More general counting variants of these problems, however, are still largely solved with hand-crafted solvers. To bridge this gap, we introduce belief propagation neural networks (BPNNs), a class of parameterized operators that operate on factor graphs and generalize Belief Propagation (BP). In its strictest form, a BPNN layer (BPNN-D) is a learned iterative operator that provably maintains many of the desirable properties of BP for any choice of the parameters. Empirically, we show that by training BPNN-D learns to perform the task better than the original BP: it converges 1. 7x faster on Ising models while providing tighter bounds. On challenging model counting problems, BPNNs compute estimates 100's of times faster than state-of-the-art handcrafted methods, while returning an estimate of comparable quality.

PDF Details

NeurIPS Conference 2020 Conference Paper

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

  • Tongzhou Mu
  • Jiayuan Gu
  • Zhiwei Jia
  • Hao Tang
  • Hao Su

We study how to learn a policy with compositional generalizability. We propose a two-stage framework, which refactorizes a high-reward teacher policy into a generalizable student policy with strong inductive bias. Particularly, we implement an object-centric GNN-based student policy, whose input objects are learned from images through self-supervised learning. Empirically, we evaluate our approach on four difficult tasks that require compositional generalizability, and achieve superior performance compared to baselines.

PDF Details

NeurIPS Conference 2020 Conference Paper

Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs

  • Hao Tang
  • Zhiao Huang
  • Jiayuan Gu
  • Bao-Liang Lu
  • Hao Su

Current graph neural networks (GNNs) lack generalizability with respect to scales (graph sizes, graph diameters, edge weights, etc. .) when solving many graph analysis problems. Taking the perspective of synthesizing graph theory programs, we propose several extensions to address the issue. First, inspired by the dependency of iteration number of common graph theory algorithms on graph size, we learn to terminate the message passing process in GNNs adaptively according to the computation progress. Second, inspired by the fact that many graph theory algorithms are homogeneous with respect to graph weights, we introduce homogeneous transformation layers that are universal homogeneous function approximators, to convert ordinary GNNs to be homogeneous. Experimentally, we show that our GNN can be trained from small-scale graphs but generalize well to large-scale graphs for a number of basic graph theory problems. It also shows generalizability for applications of multi-body physical simulation and image-based navigation problems.

PDF Details

IJCAI Conference 2016 Conference Paper

A Novel Feature Matching Strategy for Large Scale Image Retrieval

  • Hao Tang
  • Hong Liu

Feature-to-feature matching is the key issue in the Bag-of-Features model. The baseline approach employs a coarse feature-to-feature matching, namely, two descriptors are assumed to match if they are assigned the same quantization index. However, this Hard Assignment strategy usually incurs undesirable low precision. To fix it, Multiple Assignment and Soft Assignment are proposed. These two methods reduce the quantization error to some extent, but there are still a lot of room for improvement. To further improve retrieval precision, in this paper, we propose a novel feature matching strategy, called local-restricted Soft Assignment (lrSA), in which a new feature matching function is introduced. The lrSA strategy is evaluated through extensive experiments on five benchmark datasets. Experiments show that the results exceed the retrieval performance of current quantization methods on these datasets. Combined with post-processing steps, we have achieved competitive results compared with the state-of-the-art methods. Overall, our strategy shows notable benefit for retrieval with large vocabularies and dataset size.

PDF Details
v2026.09.13