Yuan Gao

EAAI Journal 2026 Journal Article

A clustering enhanced Wasserstein generative adversarial network approach for addressing uncertainty and limited data in photovoltaic output scenarios

• Qingrong Liu
• Pengfei Zhao
• Fanyue Qian
• Yuting Yao
• Hua Meng
• Yuan Gao
• Tingting Xu
• Yingjun Ruan

With the large-scale integration of photovoltaic (PV) resources introducing uncertainty and randomness to both the user and grid sides, accurately quantifying this uncertainty is critical for maintaining power grid stability and optimizing energy system operation. While scenario generation methods have advanced in addressing these challenges, they frequently rely on extensive historical data for training—an often scarce resource during the planning phase. To alleviate this limitation, this study proposes a K-medoids-enhanced Wasserstein Generative Adversarial Network (K-WGAN) integrated with feature clustering, which improves scenario generation accuracy and effectiveness and exhibits robust performance even with limited historical datasets. In comparative analyses with Latin hypercube sampling (LHS), K-WGAN (equipped with feature clustering) showed significant superiority: (1) It captured PV output characteristics more precisely, with generated scenario mean increasing by 5% and variance rising by 13% (relative to reference values); (2) As the training dataset size decreases from 100% to 4%, LHS outperforms WGAN overall under data scarcity with random sampling, especially in terms of mean and standard deviation. However, experiments using LHS for data sampling across ten groups demonstrate that WGAN exhibits superior performance, compared with random sampling; (3) Ablation experiments validated the contributions of K-medoids clustering, Wasserstein distance, and GAN structure, with the integrated model reaching 99% prediction interval coverage probability (PICP). (4) Cross-regional validation using Japan and China datasets confirmed its adaptability to diverse climates and PV systems, yielding mean deviation 7% and coverage rate 98%. These results illustrate K-WGAN supports energy system planning under data scarcity while balancing prediction accuracy and computational efficiency.

EAAI Journal 2026 Journal Article

An integrated risk assessment framework for aircraft systems based on system-theoretic process analysis and fuzzy linguistic consensus decision-making

• Baohui Jia
• Yuan Gao

As physical and functional interconnections become more complex, aircraft systems require comprehensive risk assessment. However, existing methods still face challenges, leading to inaccurate risk identification and evaluation under epistemic uncertainty. This paper proposes a framework integrating system-theoretic process analysis (STPA) with fuzzy linguistic consensus decision-making. First, we couple STPA with hesitant fuzzy linguistic term set (HFLTS), identifying the latent risk factors, and formalizing the risk factor evaluation problem. Then, the heterogeneous expert evaluations are aggregated through linguistic unification and a joint-optimized fuzzy c-means (FCM) clustering algorithm. Further, the critical risk factors are determined using a reinforcement learning-based hierarchical consensus reaching strategy, enhancing the adaptability of consensus evaluation to complex engineering scenarios, and extending the application of reinforcement learning in risk assessment. The applicability and effectiveness of the proposed framework is demonstrated by the case study of a flight control system. The results show that the proposed framework is robust and significantly improves agreement, with Kendall’s coefficient of concordance increasing from 0. 3271 to 0. 7547. Compared with existing approaches, the proposed framework achieves more accurate and consistent results, providing practical guidance for risk mitigation in the aviation industry.

TIST Journal 2026 Journal Article

Cascade Transformer for Hierarchical Semantic Reasoning in Text-Based Visual Question Answering

• Yuan Gao
• Dezhen Feng
• Laurence T. Yang
• Jing Yang
• Xiaowen Jiang
• Jieming Yang

Text-based visual question answering (TextVQA) aims to answer questions by understanding scene text in images. However, many current methods overly depend on the accuracy of Optical Character Recognition (OCR) systems, while overlooking the significance of visual objects. They tend to perform poorly when the question involves the relationships between visual objects and scene text. To address the above issues, we focus on raising the status of visual objects and innovatively propose a hierarchical semantic reasoning network (CT-HSR) based on the cascade transformer architecture, achieving fine-grained cross-modal reasoning and visual semantic enhancement. Specifically, the visual representations containing rich semantic information of the question modality are obtained through the cross-modal transformer-based vision-language pre-training model firstly. Then, the uni-modal transformer for unified modality encoding module is utilized to capture visual objects that are more semantically related to OCR texts. In addition, we further alleviate the cross-modal noise interference through the feature filtering strategy. Finally, we better align the three modalities by introducing TextVQA pre-training tasks and generate prediction answers through multi-step iterative prediction during fine-tuning. Extensive experiments on the TextVQA, ST-VQA, and OCR-VQA datasets have demonstrated the effectiveness of our proposed model compared to the state-of-the-art methods. The code will be released at https://github.com/FTFWO/CT-HSR.

AAAI Conference 2026 Conference Paper

InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer

• Muyao Yuan
• Yuanhong Zhang
• Weizhan Zhang
• Lan Ma
• Yuan Gao
• Jiangyong Ying
• Yudeng Xin

Recently, the strong generalization ability of CLIP has facilitated open-vocabulary semantic segmentation, which labels pixels using arbitrary text. However, existing methods that fine-tune CLIP for segmentation on limited seen categories often lead to overfitting and degrade the pretrained vision-language alignment. To stabilize modality alignment during fine-tuning, we propose InfoCLIP, which leverages an information-theoretic perspective to transfer alignment knowledge from pretrained CLIP to the segmentation task. Specifically, this transfer is guided by two novel objectives grounded in mutual information. First, we compress the pixel-text modality alignment from pretrained CLIP to reduce noise arising from its coarse-grained local semantic representations learned under image-text supervision. Second, we maximize the mutual information between the alignment knowledge of pretrained CLIP and the fine-tuned model to transfer compact local semantic relations suited for the segmentation task. Extensive evaluations across various benchmarks validate the effectiveness of InfoCLIP in enhancing CLIP fine-tuning for open-vocabulary semantic segmentation, demonstrating its adaptability and superiority in asymmetric transfer.

AAAI Conference 2026 Conference Paper

LUMIN: A Longitudinal Multi-modal Knowledge Decomposition Network for Predicting Breast Cancer Recurrence

• Chunyao Lu
• Tianyu Zhang
• Xinglong Liang
• Yuan Gao
• Luyi Han
• Xin Wang
• Nika Rasoolzadeh
• Tao Tan

Accurate prediction of breast cancer recurrence after treatment is essential for improving long-term outcomes. However, existing models are limited by three key challenges: (1) they typically rely on single-modal data, missing cross-modal interactions; (2) they analyze static snapshots, failing to capture disease progression over time; and (3) they often perform coarse feature fusion, lacking semantic disentanglement and interpretability. To address these issues, we propose LUMIN (Longitudinal Multi-modal Knowledge Decomposition Network), a novel framework that integrates longitudinal mammograms and electronic health records (EHRs) for recurrence prediction. LUMIN leverages a vision-language contrastive pretraining backbone to align multi-modal representations and introduces two knowledge extraction modules: (1) a Cross-Modal Disentangled Knowledge Extractor (CM-DKE) that separates shared, complementary, and modality-specific information across imaging and text; and (2) a Temporal Evolution Disentangled Knowledge Extractor (TE-DKE) that captures time-invariant, time-varying, and time-specific features to model disease dynamics. Experiments on a large-scale dataset of 3,924 patients and 19,684 exams show that LUMIN significantly outperforms state-of-the-art baselines, demonstrating its effectiveness in capturing both multi-modal semantics and temporal heterogeneity for recurrence prediction.

AAMAS Conference 2026 Conference Paper

Mind-Inference for Multi-Turn Emotional Support: Distinguishing Personal and Factual with Adaptive Memory

• Peilin Li
• Qiuyun Zhang
• Yukun Lian
• Chengxuan Li
• Bin Guo
• Mengqi Chen
• Han Wang
• Yuyang Fang

Emotional Support Conversation (ESC) requires models to track a user’s evolving psychological state and produce supportive responsesacrossmultipleturns. ExistingESCmethodsimproveempathy with emotion clues, strategy planning, or commonsense reasoning, but they often treat personal and factual information uniformly and therefore struggle with stale memories in long interactions. We present MIA, a compact framework for multi-turn ESC with three coordinated modules: EToM for hierarchical mind inference, PFD for adaptive Personal–Factual weighting, and ORM for removing obsolete psychological inferences during memory updates. Evaluations on ESConv and CPsyCounD show consistent gains over strong baselines, while ablations confirm that both adaptive factor weightingandobsolescence-awarememoryarenecessary. Theseresults indicate that explicit Personal/Factual separation is a practical design principle for robust multi-turn emotional support. ∗These authors contributed equally to this work. †Corresponding author. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). https: //doi. org/10. 65109/CIMN4023

AAAI Conference 2026 Conference Paper

NeuralOM: Neural Ocean Model for Subseasonal-to-Seasonal Simulation

• Yuan Gao
• Hao Wu
• Fan Xu
• Yanfei Xiang
• Ruijian Gou
• Ruiqi Shu
• Qingsong Wen
• Xian Wu

Long-term, high-fidelity simulation of slow-changing physical systems, such as the ocean and climate, presents a fundamental challenge in scientific computing. Traditional autoregressive machine learning models often fail in these tasks as minor errors accumulate and lead to rapid forecast degradation. To address this problem, we propose NeuralOM, a general neural operator framework designed for simulating complex, slow-changing dynamics. NeuralOM's core consists of two key innovations: (1) a Progressive Residual Correction Framework that decomposes the forecasting task into a series of fine-grained refinement steps, effectively suppressing long-term error accumulation; and (2) a Physics-Guided Graph Network whose built-in adaptive messaging mechanism explicitly models multi-scale physical interactions, such as gradient-driven flows and multiplicative couplings, thereby enhancing physical consistency while maintaining computational efficiency. We validate NeuralOM on the challenging task of global Subseasonal-to-Seasonal (S2S) ocean simulation. Extensive experiments demonstrate that NeuralOM not only surpasses state-of-the-art models in forecast accuracy and long-term stability, but also excels in simulating extreme events. For instance, at a 60-day lead time, NeuralOM achieves a 13.3% lower RMSE compared to the best-performing baseline, offering a stable, efficient, and physically-aware paradigm for data-driven scientific computing.

JBHI Journal 2026 Journal Article

SkeDiff: Skeleton 3D CT Diffusion Reconstruction using 2D X-ray

• Yuan Gao
• Rongjun Ge
• Yunbo Gu
• Zhan Wu
• Yuanhang Li
• Mingle Zhou
• Kai Chen
• Jean-Louis Coatrieux

For orthopedic diagnostics, both 2D X-ray and 3D CT imaging play essential roles. X-ray imaging is widely accessible, clinically effective, easy to operate, and has lower radiation exposure than CT. However, its inherent 2D nature limits comprehensive visualization of skeletal structures, which 3D CT provides. To bridge this gap, we propose SkeDiff, an algorithm for reconstructing 3D CT images of the skeleton from orthogonal 2D X-ray projections. To fully leverage the information in X-ray images for guiding the diffusion process, we design a cross-dimensional conditional encoder, $E\_{Cond}$, to extract 2D priors for the 3D diffusion model, $DM\_{3DL}$. This encoder integrates a CNN-Mamba hybrid architecture to enhance feature extraction and nonlinear mapping. Additionally, we introduce a 3D UKAN diffusion backbone, which employs Kolmogorov-Arnold network (KAN) to improve feature representation through learnable nonlinear activations. Furthermore, we propose a diffusion-based scoliosis classifier, $D\_{SC}$, enabling scoliosis classification during the 3D CT reconstruction process. Experiments show that SkeDiff outperforms recent algorithms on spine, hip, and knee datasets.

AAAI Conference 2026 Conference Paper

Towards Multimodal Continual Knowledge Embedding with Modality Forgetting Modulation

• Xiaowen Jiang
• Jing Yang
• ShunDong Yang
• Yuan Gao
• Xinfa Jiang
• Laurence Tianruo Yang
• Jieming Yang

The continuous emergence of new entities, relations, triples, and multimodal information drives the dynamic evolution of multimodal knowledge graph (MMKG). However, existing MMKG embedding models follow a static setting, where training from scratch for growing MMKG wastes learned knowledge, while fine-tuning on new knowledge easily leads to catastrophic forgetting, severely limiting their applicability in real-world scenarios. To address this, we propose a multimodal continual representation learning framework (MoFot) for growing MMKG. Unlike existing static multimodal embedding methods, MoFot focuses on alleviating catastrophic forgetting rather than retraining to adapt to new knowledge. Specifically, MoFot effectively mitigates catastrophic forgetting caused by parameter updates and differing forgetting rates across modalities through a multimodal collaborative modulation mechanism. The mechanism ensures consistent retention of previously learned multimodal knowledge across snapshots through multimodal weight modulation and multimodal feature modulation. MoFot outperforms existing MMKG embedding, KG continual learning, and MMKG inductive models. Experimental results demonstrate that MoFot not only avoids forgetting but also enhances old knowledge by learning new knowledge, achieving adaptation to new knowledge while mitigating forgetting of old knowledge.

TMLR Journal 2025 Journal Article

A Bias Correction Mechanism for Distributed Asynchronous Optimization

• Yuan Gao
• Yuki Takezawa
• Sebastian U Stich

We develop an asynchronous gradient method for training Machine Learning models with asynchronous distributed workers, each with its own communication and computation pace, and its own local data distribution. In the modern distributed machine learning training process, local data distribution across workers is often heterogeneous (a.k.a. client bias), which is a significant limiting factor in the analysis of most existing distributed asynchronous optimization methods. In this work, we propose AsyncBC, a distributed asynchronous variant of the SARAH method, and show that this is an effective Bias Correction mechanism for distributed asynchronous optimization. We show that AsyncBC can effectively manage arbitrary data heterogeneity, as well as handle gradient updates that arrive in an uncoordinated manner and with delays. As a byproduct of our analysis, we also provide a deeper understanding of the impacts of different stochasticity models on the convergence of the SARAH method.

ICLR Conference 2025 Conference Paper

DiffGAD: A Diffusion-based Unsupervised Graph Anomaly Detector

• Jinghan Li
• Yuan Gao
• Jinda Lu
• Junfeng Fang
• Congcong Wen
• Hui Lin
• Xiang Wang 0010

Graph Anomaly Detection (GAD) is crucial for identifying abnormal entities within networks, garnering significant attention across various fields. Traditional unsupervised methods, which decode encoded latent representations of unlabeled data with a reconstruction focus, often fail to capture critical discriminative content, leading to suboptimal anomaly detection. To address these challenges, we present a Diffusion-based Graph Anomaly Detector (DiffGAD). At the heart of DiffGAD is a novel latent space learning paradigm, meticulously designed to enhance the model's proficiency by guiding it with discriminative content. This innovative approach leverages diffusion sampling to infuse the latent space with discriminative content and introduces a content-preservation mechanism that retains valuable information across different scales, significantly improving the model’s adeptness at identifying anomalies with limited time and space complexity. Our comprehensive evaluation of DiffGAD, conducted on six real-world and large-scale datasets with various metrics, demonstrated its exceptional performance. Our code is available at https://github.com/fortunato-all/DiffGAD

EAAI Journal 2025 Journal Article

Dual-phase feature selection using adaptive neighborhood rough sets and hybrid sine-cosine optimization for classification

• Chengfeng Zheng
• Mohd Shareduwan Mohd Kasihmuddin
• Zhizhong Yan
• Mohd. Asyraf Mansor
• Yuan Gao
• Ju Chen

High-dimensional, multi-class, and imbalanced datasets present significant challenges in classification tasks across various industries, including healthcare, finance, and image processing. Existing feature selection methods, particularly those based on neighborhood rough sets, often struggle with handling both feature redundancy and noisy samples, making it difficult to capture the complex distribution of features and samples across different classes. To address this, we propose a dual-phase feature selection method that performs joint optimization in both horizontal (feature-level) and vertical (sample-level) dimensions. In the first phase, adaptive neighborhood rough set theory is used for horizontal feature selection. By adjusting the neighborhood radius ( δ ) and inclusion degree ( λ ) through cross-validation, the method selects relevant feature subsets tailored to the granularity of each dataset, thereby improving generalization. In the second phase, a hybrid sine cosine algorithm is employed for vertical processing to optimize sample selection. This algorithm iteratively removes noisy or misleading samples based on fitness evaluation, enhancing the model’s robustness. Furthermore, the framework integrates an enhanced fuzzy k-nearest neighbor classifier that leverages feature subset weights for each class to better address class imbalance during classification. Extensive experiments on 21 public datasets, using three types of classifiers, show that the proposed method outperforms seven benchmark feature selection algorithms in terms of classification accuracy, weighted precision, weighted recall, and weighted F1-score. Statistical tests, including the Wilcoxon signed-rank test, confirm significant improvements. This dual-phase horizontal and vertical optimization approach offers a robust and effective solution for real-world classification tasks involving complex data distributions.

NeurIPS Conference 2025 Conference Paper

Efficient Representativeness-Aware Coreset Selection

• Zihao Cheng
• Binrui Wu
• Zhiwei Li
• Yuesen Liao
• Su Zhao
• Shuai Chen
• Yuan Gao
• Weizhong Zhang

Dynamic coreset selection is a promising approach for improving the training efficiency of deep neural networks by periodically selecting a small subset of the most representative or informative samples, thereby avoiding the need to train on the entire dataset. However, it remains inherently challenging due not only to the complex interdependencies among samples and the evolving nature of model training, but also to a critical coreset representativeness degradation issue identified and explored in-depth in this paper, that is, the representativeness or information content of the coreset degrades over time as training progresses. Therefore, we argue that, in addition to designing accurate selection rules, it is equally important to endow the algorithms with the ability to assess the quality of the current coreset. Such awareness enables timely re-selection, mitigating the risk of overfitting to stale subsets—a limitation often overlooked by existing methods. To this end, this paper proposes an E fficient R epresentativeness- A ware C oreset S election method for deep neural networks, a lightweight framework that enables dynamic tracking and maintenance of coreset quality during training. While the ideal criterion—gradient discrepancy between the coreset and the full dataset—is computationally prohibitive, we introduce a scalable surrogate based on the signal-to-noise ratio (SNR) of gradients within the coreset, which is the main technical contribution of this paper and is also supported by our theoretical analysis. Intuitively, a decline in SNR indicates overfitting to the subset and declining representativeness. Leveraging this observation, our method triggers coreset updates without requiring costly Hessian or full-batch gradient computations, maintaining minimal computational overhead. Experiments on multiple datasets confirm the effectiveness of our approach. Notably, compared with existing gradient-based dynamic coreset selection baselines, our method achieves up to a 5. 4\% improvement in test accuracy across multiple datasets.

IROS Conference 2025 Conference Paper

Enabling On-Chip Adaptive Linear Optimal Control via Linearized Gaussian Process

• Yuan Gao
• Yinyi Lai
• Jun Wang
• Yini Fang

Unpredictable and complex aerodynamic effects pose significant challenges to achieving precise flight control, emphasizing the necessity of adaptive control via data- driven models. Moreover, real hardware usually requires high-frequency and has limited on-chip computation, making it challenging to balance the model complexity and computational cost. To address these challenges, we incorporate a linearized Gaussian process (GP) to model the external aerodynamics and combine it with linear model predictive control, enabling real-time computability. More importantly, to compensate for the control performance sacrificed by GP linearization and reduce on-chip GP computations, we design active data collection strategies using Bayesian optimization with additive GP, reducing the performance sacrifice as much as possible. Specifically, we decompose the performance into force and trajectory partitions, where the force model is for the downstream controller, and the trajectory model is used to guide collection. Experimental results show that we can achieve comparable tracking errors with full GP (not real-time computable) while maintaining real-time computable on the real Crazyflies.

ICML Conference 2025 Conference Paper

GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers

• GuoguoAi
• Guansong Pang
• Hezhe Qiao
• Yuan Gao
• Hui Yan

Graph Transformers (GTs) have demonstrated remarkable performance in graph representation learning over popular graph neural networks (GNNs). However, self-attention, the core module of GTs, preserves only low-frequency signals in graph features, leading to ineffectiveness in capturing other important signals like high-frequency ones. Some recent GT models help alleviate this issue, but their flexibility and expressiveness are still limited since the filters they learn are fixed on predefined graph spectrum or spectral order. To tackle this challenge, we propose a Graph Fourier Kolmogorov-Arnold Transformer (GrokFormer), a novel GT model that learns highly expressive spectral filters with adaptive graph spectrum and spectral order through a Fourier series modeling over learnable activation functions. We demonstrate theoretically and empirically that the proposed GrokFormer filter offers better expressiveness than other spectral methods. Comprehensive experiments on 10 real-world node classification datasets across various domains, scales, and graph properties, as well as 5 graph classification datasets, show that GrokFormer outperforms state-of-the-art GTs and GNNs. Our code is available at https: //github. com/GGA23/GrokFormer.

JBHI Journal 2025 Journal Article

MASG-SAM: Enhancing Few-Shot Medical Image Segmentation with Multi-Scale Attention and Semantic Guidance

• Wei Zhou
• Guilin Guan
• Yuan Gao
• Pengju Si
• Mengjia Xu
• Qifeng Yan

Foundation models, such as the Segment Anything Model (SAM), have demonstrated impressive generalization across various image segmentation tasks. However, they encounter challenges when applied to medical imaging, primarily due to the lack of domain-specific expertise and the limited availability of annotated data. Existing methods for adapting SAM typically rely on expert-driven prompt design and extensive fine-tuning, which hinder their effectiveness in medical imaging, particularly for rare and complex anatomical structures. To overcome these challenges, we propose MASG-SAM, an innovative framework designed for efficient few-shot medical image segmentation. MASG-SAM integrates three key innovations: the Hierarchical Attention Enhancement (HAE), Boundary Feature Enhancement (BFE), and Dynamic Semantic Fusion (DSF) modules. The HAE module optimizes attention distribution across hierarchical feature maps, enhancing feature diversity and reducing feature drift, thereby improving segmentation of both global and local features in complex medical images. The BFE module introduces a boundary-sensitive mechanism that enhances edge detection, enabling precise segmentation of overlapping or difficult-to-delineate anatomical structures. Finally, the DSF module leverages Contrastive Language-Image Pretraining (CLIP) to inject domain-specific medical semantic knowledge. By adaptively refining feature fusion during training, DSF combines semantic guidance with spatial adjustments, progressively improving segmentation accuracy, particularly in data-scarce scenarios. Experiments conducted on four publicly available medical datasets show that MASG-SAM outperforms state-of-the-art methods, achieving high segmentation accuracy with minimal labeled data. Our framework significantly enhances the adaptability and accuracy of SAM in complex medical imaging tasks. The code for MASG-SAM will be made publicly available at https://github.com/ggllllll/MASG-SAM. git.

NeurIPS Conference 2025 Conference Paper

MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query

• Wei Chow
• Yuan Gao
• Linfeng Li
• Xian Wang
• Qi Xu
• Hang Song
• Lingdong Kong
• Ran Zhou

Semantic retrieval is crucial for modern applications yet remains underexplored in current research. Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images. Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320, 000 queries with 135, 000 products in 5 languages, covering 7 distinct product categories. Extensive experiments on MERIT identify existing models's critical limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries. Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics. Experiments demonstrate that Coral achieves a 45. 9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. Collectively, our contributions—a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework—establish a foundation for future research in interleaved multi-condition semantic retrieval. Data & Code: MERIT-2025. github. io

JBHI Journal 2025 Journal Article

Multi-Modal Longitudinal Representation Learning for Predicting Neoadjuvant Therapy Response in Breast Cancer Treatment

• Yuan Gao
• Tao Tan
• Xin Wang
• Regina Beets-Tan
• Tianyu Zhang
• Luyi Han
• Antonio Portaluri
• Chunyao Lu

Longitudinal medical imaging is crucial for monitoring neoadjuvant therapy (NAT) response in clinical practice. However, mainstream artificial intelligence (AI) methods for disease monitoring commonly rely on extensive segmentation labels to evaluate lesion progression. While self-supervised vision-language (VL) learning efficiently captures medical knowledge from radiology reports, existing methods focus on single time points, missing opportunities to leverage temporal self-supervision for disease progression tracking. In addition, extracting dynamic progression from longitudinal unannotated images with corresponding textual data poses challenges. In this work, we explicitly account for longitudinal NAT examinations and accompanying reports, encompassing scans before NAT and follow-up scans during mid-/post-NAT. We introduce the multi-modal longitudinal representation learning pipeline (MLRL), a temporal foundation model, that employs multi-scale self-supervision scheme, including single-time scale vision-text alignment (VTA) learning and multi-time scale visual/textual progress (TVP/TTP) learning to extract temporal representations from each modality, thereby facilitates the downstream evaluation of tumor progress. Our method is evaluated against several state-of-the-art self-supervised longitudinal learning and multi-modal VL methods. Results from internal and external datasets demonstrate that our approach not only enhances label efficiency across the zero-, few- and full-shot regime experiments but also significantly improves tumor response prediction in diverse treatment scenarios. Furthermore, MLRL enables interpretable visual tracking of progressive areas in temporal examinations, offering insights into longitudinal VL foundation tools and potentially facilitating the temporal clinical decision-making process.

NeurIPS Conference 2025 Conference Paper

MyoChallenge 2024: A New Benchmark for Physiological Dexterity and Agility in Bionic Humans

• Huiyi Wang
• Chun Kwang Tan
• Balint Hodossy
• Shirui Lyu
• Pierre Schumacher
• James Heald
• Kai Biegun
• Samo Hromadka

Recent advancements in bionic prosthetic technology offer transformative opportunities to restore mobility and functionality for individuals with missing limbs. Users of bionic limbs, or bionic humans, learn to seamlessly integrate prosthetic extensions into their motor repertoire, regaining critical motor abilities. The remarkable movement generalization and environmental adaptability demonstrated by these individuals highlight motor intelligence capabilities unmatched by current artificial intelligence systems. Addressing these limitations, MyoChallenge '24 at NeurIPS 2024 established a benchmark for human-robot coordination with an emphasis on joint control of both biological and mechanical limbs. The competition featured two distinct tracks: a manipulation task utilizing the myoMPL model, integrating a virtual biological arm and the Modular Prosthetic Limb (MPL) for a passover task; and a locomotion task using the novel myoOSL model, combining a bilateral virtual biological leg with a trans-femoral amputation and the Open Source Leg (OSL) to navigate varied terrains. Marking the third iteration of the MyoChallenge, the event attracted over 50 teams with more than 290 submissions all around the globe, with diverse participants ranging from independent researchers to high school students. The competition facilitated the development of several state-of-the-art control algorithms for bionic musculoskeletal systems, leveraging techniques such as imitation learning, muscle synergy, and model-based reinforcement learning that significantly surpassed our proposed baseline performance by a factor of 10. By providing the open-source simulation framework of MyoSuite, standardized tasks, and physiologically realistic models, MyoChallenge serves as a reproducible testbed and benchmark for bridging ML and biomechanics. The competition website is featured here: https: //sites. google. com/view/myosuite/myochallenge/myochallenge-2024.

AAAI Conference 2025 Conference Paper

On the Convergence of Tâtonnement for Linear Fisher Markets

• Tianlong Nan
• Yuan Gao
• Christian Kroer

Tâtonnement is a simple, intuitive market process where prices are iteratively adjusted based on the difference between demand and supply. Many variants under different market assumptions have been studied and shown to converge to a market equilibrium, in some cases at a fast rate. However, the classical case of linear Fisher markets have long eluded the analyses, and it remains unclear whether tâtonnement converges in this case. We show that, for a sufficiently small stepsize, the prices given by the tâtonnement process are guaranteed to converge to equilibrium prices, up to a small approximation radius that depends on the stepsize. To achieve this, we consider the dual Eisenberg-Gale convex program in the price space, view tâtonnement as subgradient descent on this convex program, and utilize novel last-iterate convergence results for subgradient descent under error bound conditions. In doing so, we show that the convex program satisfies a particular error bound condition, the quadratic growth condition, and that the price sequence generated by tâtonnement is bounded above and away from zero. We also show that a similar convergence result holds for tâtonnement in quasi-linear Fisher markets. Numerical experiments are conducted to demonstrate that the theoretical linear convergence aligns with empirical observations.

ICML Conference 2025 Conference Paper

OneForecast: A Universal Framework for Global and Regional Weather Forecasting

• Yuan Gao
• Hao Wu 0094
• Ruiqi Shu
• Huanshuo Dong
• Fan Xu 0009
• Rui Ray Chen
• Yibo Yan
• Qingsong Wen

Accurate weather forecasts are important for disaster prevention, agricultural planning, etc. Traditional numerical weather prediction (NWP) methods offer physically interpretable high-accuracy predictions but are computationally expensive and fail to fully leverage rapidly growing historical data. In recent years, deep learning models have made significant progress in weather forecasting, but challenges remain, such as balancing global and regional high-resolution forecasts, excessive smoothing in extreme event predictions, and insufficient dynamic system modeling. To address these issues, this paper proposes a global-regional nested weather forecasting framework (OneForecast) based on graph neural networks. By combining a dynamic system perspective with multi-grid theory, we construct a multi-scale graph structure and densify the target region to capture local high-frequency features. We introduce an adaptive messaging mechanism, using dynamic gating units to deeply integrate node and edge features for more accurate extreme event forecasting. For high-resolution regional forecasts, we propose a neural nested grid method to mitigate boundary information loss. Experimental results show that OneForecast performs excellently across global to regional scales and short-term to long-term forecasts, especially in extreme event predictions. Codes link: https: //github. com/YuanGao-YG/OneForecast.

JBHI Journal 2025 Journal Article

scSwinTNet: A Cell Type Annotation Method for Large-Scale Single-Cell RNA-Seq Data Based on Shifted Window Attention

• Huanhuan Dai
• Xiangyu Meng
• Zhiyi Pan
• Qing Yang
• Haonan Song
• Yuan Gao
• Xun Wang

The annotation of cell types based on single-cell RNA sequencing (scRNA-seq) data is a critical downstream task in single-cell analysis, with significant implications for a deeper understanding of biological processes. Most analytical methods cluster cells by unsupervised clustering, which requires manual annotation for cell type determination. This procedure is time-overwhelming and non-repeatable. To accommodate the exponential growth of sequencing cells, reduce the impact of data bias, and integrate large-scale datasets for further improvement of type annotation accuracy, we proposed scSwinTNet. It is a pre-trained tool for annotating cell types in scRNA-seq data, which uses self-attention based on shifted windows and enables intelligent information extraction from gene data. We demonstrated the effectiveness and robustness of scSwinTNet by using 399 760 cells from human and mouse tissues. To the best of our knowledge, scSwinTNet is the first model to annotate cell types in scRNA-seq data using a pre-trained shifted window attention-based model. It does not require a priori knowledge and accurately annotates cell types without manual annotation.

EAAI Journal 2025 Journal Article

Three-dimensional reconstruction and fracture segmentation based on X-ray and computed tomography paired dataset

• Yuan Gao
• Yuan Zhou
• Da Chen
• Jiachen Li
• Mingle Zhou
• Gang Li
• Yunbo Gu
• Jean-Louis Coatrieux

In some orthopedic surgeries, the use of three-dimensional (3D) computed tomography (CT) scanning technology is not feasible due to scene limitations, leaving doctors to rely on two-dimensional (2D) X-ray images for real-time diagnosis. However, X-ray images lack 3D information, making accurate diagnosis challenging. Developing an algorithm to convert 2D X-ray images into 3D CT images, while simultaneously combining high-quality 3D reconstruction with precise fracture segmentation, offers a promising solution to the problem. In this study, we propose a novel artificial intelligence (AI)-driven framework named 3D reconstruction and segment anything model (3DRecSAM). The reconstruction image enhancer (RIE) is designed to achieve high-precision 3D reconstruction and provide high-quality feature initialization for fracture segmentation. Meanwhile, the mamba segment anything model (MSAM), based on the segment anything model (SAM) architecture, is developed for accurate fracture segmentation. We introduce a Kolmogorov–Arnold network (KAN)-based attention fusion module (KAF), which facilitates the joint optimization of the RIE reconstruction network and the MSAM segmentation network. Furthermore, the selective scanning mamba with KAN (SKM) is incorporated to enhance feature extraction for both RIE and MSAM. Mamba efficiently captures long-range dependencies and sequential patterns, while KAN’s learnable activation functions facilitate adaptive feature fusion and non-linear representation. To train and evaluate 3DRecSAM, we introduce the real X-ray and CT paired dataset (XCPData), which is publicly available on GitHub: https: //github. com/YuanGao1201/XCPData.

TAAS Journal 2024 Journal Article

Dynamically Balancing Load with Overload Control for Microservices

• Ratnadeep Bhattacharya
• Yuan Gao
• Timothy Wood

The microservices architecture simplifies application development by breaking monolithic applications into manageable microservices. However, this distributed microservice “service mesh” leads to new challenges due to the more complex application topology. Particularly, each service component scales up and down independently creating load imbalance problems on shared backend services accessed by multiple components. Traditional load balancing algorithms do not port over well to a distributed microservice architecture where load balancers are deployed client-side. In this article, we propose a self-managing load balancing system, BLOC, which provides consistent response times to users without using a centralized metadata store or explicit messaging between nodes. BLOC uses overload control approaches to provide feedback to the load balancers. We show that this performs significantly better in solving the incast problem in microservice architectures. A critical component of BLOC is the dynamic capacity estimation algorithm. We show that a well-tuned capacity estimate can outperform even join-the-shortest-queue, a nearly optimal algorithm, while a reasonable dynamic estimate still outperforms Least Connection, a distributed implementation of join-the-shortest-queue. Evaluating this framework, we found that BLOC improves the response time distribution range, between the 10th and 90th percentiles, by 2 –4 times and the tail, 99th percentile, latency by 2 times.

JMLR Journal 2024 Journal Article

Gaussian Interpolation Flows

• Yuan Gao
• Jian Huang
• Yuling Jiao

Gaussian denoising has emerged as a powerful method for constructing simulation-free continuous normalizing flows for generative modeling. Despite their empirical successes, theoretical properties of these flows and the regularizing effect of Gaussian denoising have remained largely unexplored. In this work, we aim to address this gap by investigating the well-posedness of simulation-free continuous normalizing flows built on Gaussian denoising. Through a unified framework termed Gaussian interpolation flow, we establish the Lipschitz regularity of the flow velocity field, the existence and uniqueness of the flow, and the Lipschitz continuity of the flow map and the time-reversed flow map for several rich classes of target distributions. This analysis also sheds light on the auto-encoding and cycle consistency properties of Gaussian interpolation flows. Additionally, we study the stability of these flows in source distributions and perturbations of the velocity field, using the quadratic Wasserstein distance as a metric. Our findings offer valuable insights into the learning techniques employed in Gaussian interpolation flows for generative modeling, providing a solid theoretical foundation for end-to-end error analyses of learning Gaussian interpolation flows with empirical observations. [abs] [ pdf ][ bib ] &copy JMLR 2024. ( edit, beta )

AAAI Conference 2024 Conference Paper

Locality Preserving Refinement for Shape Matching with Functional Maps

• Yifan Xia
• Yifan Lu
• Yuan Gao
• Jiayi Ma

In this paper, we address the nonrigid shape matching with outliers by a novel and effective pointwise map refinement method, termed Locality Preserving Refinement. For accurate pointwise conversion from a given functional map, our method formulates a two-step procedure. Firstly, starting with noisy point-to-point correspondences, we identify inliers by leveraging the neighborhood support, which yields a closed-form solution with linear time complexity. After obtained the reliable correspondences of inliers, we refine the pointwise correspondences for outliers using local linear embedding, which operates in an adaptive spectral similarity space to further eliminate the ambiguities that are difficult to handle in the functional space. By refining pointwise correspondences with local consistency thus embedding geometric constraints into functional spaces, our method achieves considerable improvement in accuracy with linearithmic time and space cost. Extensive experiments on public benchmarks demonstrate the superiority of our method over the state-of-the-art methods. Our code is publicly available at https://github.com/XiaYifan1999/LOPR.

EAAI Journal 2024 Journal Article

Progressive structure enhancement graph convolutional network for face clustering

• Shaoying Li
• Wei Yao
• Yuan Gao
• Yinchi Ma
• Bo Liu

Face clustering, a technique for automatically annotating large-scale face data, has made significant advancements with the advent of graph convolutional networks (GCNs). Despite their success, GCNs can suffer from decreased performance due to conflicting information passed along noisy edges of a graph. To address this issue, we propose a novel framework named progressive structure enhancement GCN (PSE-GCN), which combines graph structure learning with graph-guided feature aggregation. Our PSE-GCN framework includes a dynamic graph construction (DGC) module that enhances local relationships and suppresses global noise, thereby improving the quality of the graph. By stacking multiple DGCs, PSE-GCN progressively refines the graph quality and yields discriminative features for various clustering tasks. Additionally, we introduce a subgraph-based neighborhood re-ranking (SNR) mechanism that improves graph homogeneity by rearranging the candidate neighbors of each face based on structural similarity at the subgraph level. Our experimental results, conducted on several popular benchmarks, not only demonstrate the effectiveness of PSE-GCN, but also show that it outperforms state-of-the-art methods, e. g. , 93. 50% in pairwise F-score on the MS-Celeb-1M dataset.

NeurIPS Conference 2024 Conference Paper

Towards Neuron Attributions in Multi-Modal Large Language Models

• Junfeng Fang
• Zongze Bi
• Ruipeng Wang
• Houcheng Jiang
• Yuan Gao
• Kun Wang
• An Zhang
• Jie Shi

As Large Language Models (LLMs) demonstrate impressive capabilities, demystifying their internal mechanisms becomes increasingly vital. Neuron attribution, which attributes LLM outputs to specific neurons to reveal the semantic properties they learn, has emerged as a key interpretability approach. However, while neuron attribution has made significant progress in deciphering text-only LLMs, its application to Multimodal LLMs (MLLMs) remains less explored. To address this gap, we propose a novel Neuron Attribution method tailored for MLLMs, termed NAM. Specifically, NAM not only reveals the modality-specific semantic knowledge learned by neurons within MLLMs, but also highlights several intriguing properties of neurons, such as cross-modal invariance and semantic sensitivity. These properties collectively elucidate the inner workings mechanism of MLLMs, providing a deeper understanding of how MLLMs process and generate multi-modal content. Through theoretical analysis and empirical validation, we demonstrate the efficacy of NAM and the valuable insights it offers. Furthermore, leveraging NAM, we introduce a multi-modal knowledge editing paradigm, underscoring the practical significance of our approach for downstream applications of MLLMs.

ICML Conference 2024 Conference Paper

Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding

• Guangyi Liu 0005
• Yu Wang 0170
• Zeyu Feng
• Qiyu Wu 0001
• Liping Tang
• Yuan Gao
• Zhen Li 0026
• Shuguang Cui

The vast applications of deep generative models are anchored in three core capabilities— generating new instances, reconstructing inputs, and learning compact representations —across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce Generalized E ncoding - D ecoding D iffusion P robabilistic M odels (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e. g. , large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models. Code is available at https: //github. com/guangyliu/EDDPM.

NeurIPS Conference 2024 Conference Paper

VideoTetris: Towards Compositional Text-to-Video Generation

• Ye Tian
• Ling Yang
• Haotian Yang
• Yuan Gao
• Yufan Deng
• Jingmin Chen
• Xintao Wang
• Zhaochen Yu

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose a new dynamic-aware data processing pipeline and a consistency regularization method to enhance the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https: //github. com/YangLing0818/VideoTetris

NeurIPS Conference 2023 Conference Paper

Evaluating Post-hoc Explanations for Graph Neural Networks via Robustness Analysis

• Junfeng Fang
• Wei Liu
• Yuan Gao
• Zemin Liu
• An Zhang
• Xiang Wang
• Xiangnan He

This work studies the evaluation of explaining graph neural networks (GNNs), which is crucial to the credibility of post-hoc explainability in practical usage. Conventional evaluation metrics, and even explanation methods -- which mainly follow the paradigm of feeding the explanatory subgraph and measuring output difference -- always suffer from the notorious out-of-distribution (OOD) issue. In this work, we endeavor to confront the issue by introducing a novel evaluation metric, termed O OD-resistant A dversarial R obustness (OAR). Specifically, we draw inspiration from the notion of adversarial robustness and evaluate post-hoc explanation subgraphs by calculating their robustness under attack. On top of that, an elaborate OOD reweighting block is inserted into the pipeline to confine the evaluation process to the original data distribution. For applications involving large datasets, we further devise a Sim plified version of OAR (SimOAR), which achieves a significant improvement in computational efficiency at the cost of a small amount of performance. Extensive empirical studies validate the effectiveness of our OAR and SimOAR.

AAAI Conference 2023 Conference Paper

Exploit Domain-Robust Optical Flow in Domain Adaptive Video Semantic Segmentation

• Yuan Gao
• Zilei Wang
• Jiafan Zhuang
• Yixin Zhang
• Junjie Li

Domain adaptive semantic segmentation aims to exploit the pixel-level annotated samples on source domain to assist the segmentation of unlabeled samples on target domain. For such a task, the key is to construct reliable supervision signals on target domain. However, existing methods can only provide unreliable supervision signals constructed by segmentation model (SegNet) that are generally domain-sensitive. In this work, we try to find a domain-robust clue to construct more reliable supervision signals. Particularly, we experimentally observe the domain-robustness of optical flow in video tasks as it mainly represents the motion characteristics of scenes. However, optical flow cannot be directly used as supervision signals of semantic segmentation since both of them essentially represent different information. To tackle this issue, we first propose a novel Segmentation-to-Flow Module (SFM) that converts semantic segmentation maps to optical flows, named the segmentation-based flow (SF), and then propose a Segmentation-based Flow Consistency (SFC) method to impose consistency between SF and optical flow, which can implicitly supervise the training of segmentation model. The extensive experiments on two challenging benchmarks demonstrate the effectiveness of our method, and it outperforms previous state-of-the-art methods with considerable performance improvement. Our code is available at https://github.com/EdenHazardan/SFC.

AAAI Conference 2023 Conference Paper

Fast and Interpretable Dynamics for Fisher Markets via Block-Coordinate Updates

• Tianlong Nan
• Yuan Gao
• Christian Kroer

We consider the problem of large-scale Fisher market equilibrium computation through scalable first-order optimization methods. It is well-known that market equilibria can be captured using structured convex programs such as the Eisenberg-Gale and Shmyrev convex programs. Highly performant deterministic full-gradient first-order methods have been developed for these programs. In this paper, we develop new block-coordinate first-order methods for computing Fisher market equilibria, and show that these methods have interpretations as tâtonnement-style or proportional response-style dynamics where either buyers or items show up one at a time. We reformulate these convex programs and solve them using proximal block coordinate descent methods, a class of methods that update only a small number of coordinates of the decision variable in each iteration. Leveraging recent advances in the convergence analysis of these methods and structures of the equilibrium-capturing convex programs, we establish fast convergence rates of these methods.

NeurIPS Conference 2023 Conference Paper

Inferring Hybrid Neural Fluid Fields from Videos

• Hong-Xing Yu
• Yang Zheng
• Yuan Gao
• Yitong Deng
• Bo Zhu
• Jiajun Wu

We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https: //kovenyu. com/HyFluid/

ICLR Conference 2023 Conference Paper

Statistical Inference for Fisher Market Equilibrium

• Luofeng Liao
• Yuan Gao
• Christian Kroer

Statistical inference under market equilibrium effects has attracted increasing attention recently. In this paper we focus on the specific case of linear Fisher markets. They have been widely use in fair resource allocation of food/blood donations and budget management in large-scale Internet ad auctions. In resource allocation, it is crucial to quantify the variability of the resource received by the agents (such as blood banks and food banks) in addition to fairness and efficiency properties of the systems. For ad auction markets, it is important to establish statistical properties of the platform's revenues in addition to their expected values. To this end, we propose a statistical framework based on the concept of infinite-dimensional Fisher markets. In our framework, we observe a market formed by a finite number of items sampled from an underlying distribution (the ``observed market'') and aim to infer several important equilibrium quantities of the underlying long-run market. These equilibrium quantities include individual utilities, social welfare, and pacing multipliers. Through the lens of sample average approximation (SSA), we derive a collection of statistical results and show that the observed market provides useful statistical information of the long-run market. In other words, the equilibrium quantities of the observed market converge to the true ones of the long-run market with strong statistical guarantees. These include consistency, finite sample bounds, asymptotics, and confidence. As an extension, we discuss revenue inference in quasilinear Fisher markets.

JBHI Journal 2022 Journal Article

Interpretability Analysis of One-Year Mortality Prediction for Stroke Patients Based on Deep Neural Network

• Shuo Zhang
• Jing Wang
• Lulu Pei
• Kai Liu
• Yuan Gao
• Hui Fang
• Rui Zhang
• Lu Zhao

Clinically, physicians collect the benchmark medical data to establish archives for a stroke patient and then add the follow up data regularly. It has great significance on prognosis prediction for stroke patients. In this paper, we present an interpretable deep learning model to predict the one-year mortality risk on stroke. We design sub-modules to reconstruct features from original clinical data that highlight the dissimilarity and temporality of different variables. The model consists of Bidirectional Long Short-Term Memory (Bi-LSTM), in which a novel correlation attention module is proposed that takes the correlation of variables into consideration. In experiments, datasets are collected clinically from the department of neurology in a local AAA hospital. It consists of 2, 275 stroke patients hospitalized in the department of neurology from 2014 to 2016. Our model achieves a precision of 0. 9414, a recall of 0. 9502 and an F1-score of 0. 9415. In addition, we provide the analysis of the interpretability by visualizations with reference to clinical professional guidelines.

IJCAI Conference 2022 Conference Paper

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

• Min Peng
• Chongyang Wang
• Yuan Gao
• Yu Shi
• Xiang-Dong Zhou

Video question answering (VideoQA) is challenging given its multimodal combination of visual understanding and natural language processing. While most existing approaches ignore the visual appearance-motion information at different temporal scales, it is unknown how to incorporate the multilevel processing capacity of a deep learning model with such multiscale information. Targeting these issues, this paper proposes a novel Multilevel Hierarchical Network (MHN) with multiscale sampling for VideoQA. MHN comprises two modules, namely Recurrent Multimodal Interaction (RMI) and Parallel Visual Reasoning (PVR). With a multiscale sampling, RMI iterates the interaction of appearance-motion information at each scale and the question embeddings to build the multilevel question-guided visual representations. Thereon, with a shared transformer encoder, PVR infers the visual cues at each level in parallel to fit with answering different question types that may rely on the visual information at relevant levels. Through extensive experiments on three VideoQA datasets, we demonstrate improved performances than previous state-of-the-arts and justify the effectiveness of each part of our method.

NeurIPS Conference 2022 Conference Paper

Nonstationary Dual Averaging and Online Fair Allocation

• Luofeng Liao
• Yuan Gao
• Christian Kroer

We consider the problem of fairly allocating sequentially arriving items to a set of individuals. For this problem, the recently-introduced PACE algorithm leverages the dual averaging algorithm to approximate competitive equilibria and thus generate online fair allocations. PACE is simple, distributed, and parameter-free, making it appealing for practical use in large-scale systems. However, current performance guarantees for PACE require i. i. d. item arrivals. Since real-world data is rarely i. i. d. , or even stationary, we study the performance of PACE on nonstationary data. We start by developing new convergence results for the general dual averaging algorithm under three nonstationary input models: adversarially-corrupted stochastic input, ergodic input, and block-independent (including periodic) input. Our results show convergence of dual averaging up to errors caused by nonstationarity of the data, and recover the classical bounds when the input data is i. i. d. Using these results, we show that the PACE algorithm for online fair allocation simultaneously achieves ``best of many worlds'' guarantees against any of these nonstationary input models as well as against i. i. d. input. Finally, numerical experiments show strong empirical performance of PACE against nonstationary inputs.

YNIMG Journal 2022 Journal Article

Spatial-topographic nestedness of interoceptive regions within the networks of decision making and emotion regulation: Combining ALE meta-analysis and MACM analysis

• Yafei Tan
• Ranran Yan
• Yuan Gao
• Meng Zhang
• Georg Northoff

Prominent theories propose that interoception modulates our behavioral and emotional responses involving decision-making and emotion regulation. Are the regions implicated in interoception also spatially related to and possibly nested within the networks of decision making and emotion regulation? Addressing this question, we performed three meta-analyses of functional magnetic resonance imaging studies to identify the regions that are commonly activated by the three domains using activation likelihood estimation (ALE). Additionally, we assessed the coactivation pattern of identified common regions using meta-analytic connectivity modeling (MACM). The results showed major overlaps of interoception with both decision making and emotion regulation in specifically the right dorsal anterior insula. The pairwise contrast analyses confirmed this finding and revealed conjunction-based activities in decision making and emotion regulation in the dorsal anterior cingulate cortex (dACC). MACM based on the identified insula revealed a widespread convergent coactivation pattern with the left anterior insula, dACC, and bilateral thalamus which, together, constitute the salience network. Among these co-activated regions, bilateral insula and the dACC were shared among all three domains. These results suggest that the regions mediating interoception including intero-exteroceptive integration and salience attribution are contained and thus spatially nested within the more extensive networks recruited during decision making and emotion regulation.

AAAI Conference 2021 Conference Paper

Exploiting Learnable Joint Groups for Hand Pose Estimation

• Moran Li
• Yuan Gao
• Nong Sang

In this paper, we propose to estimate 3D hand pose by recovering the 3D coordinates of joints in a group-wise manner, where less-related joints are automatically categorized into different groups and exhibit different features. This is different from the previous methods where all the joints are considered holistically and share the same feature. The benefits of our method are illustrated by the principle of multi-task learning (MTL), i. e. , by separating less-related joints into different groups (as different tasks), our method learns different features for each of them, therefore efficiently avoids the negative transfer (among less related tasks/groups of joints). The key of our method is a novel binary selector that automatically selects related joints into the same group. We implement such a selector with binary values stochastically sampled from a Concrete distribution, which is constructed using Gumbel softmax on trainable parameters. This enables us to preserve the differentiable property of the whole network. We further exploit features from those less-related groups by carrying out an additional feature fusing scheme among them, to learn more discriminative features. This is realized by implementing multiple 1x1 convolutions on the concatenated features, where each joint group contains a unique 1x1 convolution for feature fusion. The detailed ablation analysis and the extensive experiments on several benchmark datasets demonstrate the promising performance of the proposed method over the state-of-the-art (SOTA) methods. Besides, our method achieves top-1 among all the methods that do not exploit the dense 3D shape labels on the most recently released FreiHAND competition at the submission date. The source code and models are available at https: //github. com/ moranli-aca/LearnableGroups-Hand.

AAAI Conference 2021 Conference Paper

Increasing Iterate Averaging for Solving Saddle-Point Problems

• Yuan Gao
• Christian Kroer
• Donald Goldfarb

Many problems in machine learning and game theory can be formulated as saddle-point problems, for which various first-order methods have been developed and proven efficient in practice. Under the general convex-concave assumption, most first-order methods only guarantee an ergodic convergence rate, that is, the uniform averages of the iterates converge at a O(1/T) rate in terms of the saddle-point residual. However, numerically, the iterates themselves can often converge much faster than the uniform averages. This observation motivates increasing averaging schemes that put more weight on later iterates, in contrast to the usual uniform averaging. We show that such increasing averaging schemes, applied to various first-order methods, are able to preserve the O(1/T) convergence rate with no additional assumptions or computational overhead. Extensive numerical experiments on zero-sum game solving, market equilibrium computation and image denoising demonstrate the effectiveness of the proposed schemes. In particular, the increasing averages consistently outperform the uniform averages in all test problems by orders of magnitude. When solving matrix and extensiveform games, increasing averages consistently outperform the last iterates as well. For matrix games, a first-order method equipped with increasing averaging outperforms the highly competitive CFR+ algorithm.

AAAI Conference 2021 Conference Paper

Infinite-Dimensional Fisher Markets: Equilibrium, Duality and Optimization

• Yuan Gao
• Christian Kroer

This paper considers a linear Fisher market with n buyers and a continuum of items. In order to compute market equilibria, we introduce (infinite-dimensional) convex programs over Banach spaces, thereby generalizing the Eisenberg-Gale convex program and its dual. Regarding the new convex programs, we establish existence of optimal solutions, KKT conditions, as well as strong duality. All these properties are established via non-standard arguments, which circumvent the limitations of duality theory in optimization over infinitedimensional vector spaces. Furthermore, we show that there exists a pure equilibrium allocation, i. e. , a division of the item space. Similar to the finite-dimensional case, a market equilibrium under the infinite-dimensional Fisher market is Pareto optimal, envy-free and proportional. We also show how to obtain the (a. e. unique) equilibrium prices and a pure equilibrium allocation from the (unique) equilibrium utility prices. When the item space is the unit interval [0, 1] and buyers have piecewise linear utilities, we show that approximate equilibrium prices can be computed in polynomial time. This is achieved by solving a finite-dimensional convex program using the ellipsoid method. To this end, we give nontrivial and efficient subgradient and separation oracles. For general buyer valuations, we propose computing market equilibrium using stochastic dual averaging, which finds approximate equilibrium prices with high probability.

NeurIPS Conference 2021 Conference Paper

Online Market Equilibrium with Application to Fair Division

• Yuan Gao
• Alex Peysakhovich
• Christian Kroer

Computing market equilibria is a problem of both theoretical and applied interest. Much research to date focuses on the case of static Fisher markets with full information on buyers' utility functions and item supplies. Motivated by real-world markets, we consider an online setting: individuals have linear, additive utility functions; items arrive sequentially and must be allocated and priced irrevocably. We define the notion of an online market equilibrium in such a market as time-indexed allocations and prices which guarantee buyer optimality and market clearance in hindsight. We propose a simple, scalable and interpretable allocation and pricing dynamics termed as PACE. When items are drawn i. i. d. from an unknown distribution (with a possibly continuous support), we show that PACE leads to an online market equilibrium asymptotically. In particular, PACE ensures that buyers' time-averaged utilities converge to the equilibrium utilities w. r. t. a static market with item supplies being the unknown distribution and that buyers' time-averaged expenditures converge to their per-period budget. Hence, many desirable properties of market equilibrium-based fair division such as envy-freeness, Pareto optimality, and the proportional-share guarantee are also attained asymptotically in the online setting. Next, we extend the dynamics to handle quasilinear buyer utilities, which gives the first online algorithm for computing first-price pacing equilibria. Finally, numerical experiments on real and synthetic datasets show that the dynamics converges quickly under various metrics.

NeurIPS Conference 2020 Conference Paper

An Improved Analysis of Stochastic Gradient Descent with Momentum

• Yanli Liu
• Yuan Gao
• Wotao Yin

SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of momentum is still unclear in general since previous analyses on SGDM either provide worse convergence bounds than those of SGD, or assume Lipschitz or quadratic objectives, which fail to hold in practice. Furthermore, the role of dynamic parameters has not been addressed. In this work, we show that SGDM converges as fast as SGD for smooth objectives under both strongly convex and nonconvex settings. We also prove that multistage strategy is beneficial for SGDM compared to using fixed parameters. Finally, we verify these theoretical claims by numerical experiments.

NeurIPS Conference 2020 Conference Paper

First-Order Methods for Large-Scale Market Equilibrium Computation

• Yuan Gao
• Christian Kroer

Market equilibrium is a solution concept with many applications such as digital ad markets, fair division, and resource sharing. For many classes of utility functions, equilibria can be captured by convex programs. We develop simple first-order methods suitable for solving these programs for large-scale markets. We focus on three practically-relevant utility classes: linear, quasilinear, and Leontief utilities. Using structural properties of market equilibria under each utility class, we show that the corresponding convex programs can be reformulated as optimization of a structured smooth convex function over a polyhedral set, for which projected gradient achieves linear convergence. To do so, we utilize recent linear convergence results under weakened strong-convexity conditions, and further refine the relevant constants in existing convergence results. Then, we show that proximal gradient (a generalization of projected gradient) with a practical linesearch scheme achieves linear convergence under the Proximal-PL condition, a recently developed error bound condition for convex composite problems. For quasilinear utilities, we show that Mirror Descent applied to a new convex program achieves sublinear last-iterate convergence and yields a form of Proportional Response dynamics, an elegant, interpretable algorithm for computing market equilibria originally developed for linear utilities. Numerical experiments show that Proportional Response is highly efficient for computing approximate market equilibria, while projected gradient with linesearch can be much faster when higher-accuracy solutions are needed.

AAAI Conference 2018 Conference Paper

Reduced-Rank Linear Dynamical Systems

• Qi She
• Yuan Gao
• Kai Xu
• Rosa Chan

Linear Dynamical Systems are widely used to study the underlying patterns of multivariate time series. A basic assumption of these models is that high-dimensional time series can be characterized by some underlying, low-dimensional and time-varying latent states. However, existing approaches to LDS modeling mostly learn the latent space with a prescribed dimensionality. When dealing with short-length highdimensional time series data, such models would be easily overfitted. We propose Reduced-Rank Linear Dynamical Systems (RRLDS), to automatically retrieve the intrinsic dimensionality of the latent space during model learning. Our key observation is that the rank of the dynamics matrix of LDS captures the intrinsic dimensionality, and the variational inference with a reduced-rank regularization finally leads to a concise, structured, and interpretable latent space. To enable our method to handle count-valued data, we introduce the dispersion-adaptive distribution to accommodate over-/ equal- / and under-dispersion nature of such data. Results on both simulated and experimental data demonstrate our model can robustly learn latent space from short-length, noisy, countvalued data and significantly improve the prediction performance over the state-of-the-art methods.

IJCAI Conference 2017 Conference Paper

Locality Preserving Matching

• Jiayi Ma
• Ji Zhao
• Hanqi Guo
• Junjun Jiang
• Huabing Zhou
• Yuan Gao

Seeking reliable correspondences between two feature sets is a fundamental and important task in computer vision. This paper attempts to remove mismatches from given putative image feature correspondences. To achieve the goal, an efficient approach, termed as locality preserving matching (LPM), is designed, the principle of which is to maintain the local neighborhood structures of those potential true matches. We formulate the problem into a mathematical model, and derive a closed-form solution with linearithmic time and linear space complexities. More specifically, our method can accomplish the mismatch removal from thousands of putative correspondences in only a few milliseconds. Experiments on various real image pairs for general feature matching, as well as for visual homing and image retrieval demonstrate the generality of our method for handling different types of image deformations, and it is more than two orders of magnitude faster than state-of-the-art methods in the same range of or better accuracy.

EAAI Journal 2017 Journal Article

Machining vibration states monitoring based on image representation using convolutional neural networks

• Yang Fu
• Yun Zhang
• Yuan Gao
• Huang Gao
• Ting Mao
• Huamin Zhou
• Dequn Li

Measured signals are usually fed into filters or signal decomposers to extract useful features to assist making identification in state monitoring or fault diagnosis. But what is routinely ignored is that an experienced expert can realize what is happening just by watching the signals presented on the oscilloscope even without the analyzing report. The vision image input and the experience feedback are the two keys in this identification process by the brain. The experience can be easily quantified, like 1 for “good” and 0 for “bad”, and used for identification model construction, while there has been no attempt to use pictured signal as the model input. For closed-loop control system, it is necessary to acquire signal feedback point by point to adjust the system in real time. But for state monitoring and fault diagnosis, the pattern hiding among the signal points is usually more important, which is exactly one of the special fields of image representation to indicate complex interrelationship. Taking machining state monitoring as example, this paper explore the possibility to use the pictured signals as input to construct identification model without traditional feature engineering based on signal analysis. Convolutional neural networks (CNN) is introduced to connect pictured signals to different vibration states with experience feedback. Results validate the proposed method with excellent modeling performance. Time complexity analysis proves this pictured signal image representation based CNN method to be capable to be real-time. Two dimensional image representation is a powerful way to exhibit and fuse information. With high flexibility, the proposed method may be a promising framework for monitoring or fault diagnosis tasks.

TCS Journal 2013 Journal Article

State complexity of star of union and square of union on k regular languages

• Yuan Gao
• Lila Kari

In this paper, we investigate the state complexities of ( ⋃ i = 1 k L i ) ⁎ and ( ⋃ i = 1 k L i ) 2, where L i, 1 ⩽ i ⩽ k, k ⩾ 2 are regular languages. We establish exact bounds for both of these general combined operations and show that they are much lower than the mathematical compositions of the state complexities of their basic individual component operations, but have similar forms with the state complexities of some participating combined operations.

TCS Journal 2012 Journal Article

State complexity of combined operations with two basic operations

• Bo Cui
• Yuan Gao
• Lila Kari
• Sheng Yu

This paper studies the state complexity of ( L 1 L 2 ) R, L 1 R L 2, L 1 ∗ L 2, ( L 1 ∪ L 2 ) L 3, ( L 1 ∩ L 2 ) L 3, L 1 L 2 ∩ L 3, and L 1 L 2 ∪ L 3 for regular languages L 1, L 2, and L 3. We first show that the upper bound proposed by Liu et al. (2008) [18] for the state complexity of ( L 1 L 2 ) R coincides with the lower bound and is thus the state complexity of this combined operation by providing some witness DFAs. Also, we show that, unlike most other cases, due to the structural properties of the result of the first operation of the combinations L 1 R L 2, L 1 ∗ L 2, and ( L 1 ∪ L 2 ) L 3, the state complexity of each of these combined operations is close to the mathematical composition of the state complexities of the component operations. Moreover, we show that the state complexities of ( L 1 ∩ L 2 ) L 3, L 1 L 2 ∩ L 3, and L 1 L 2 ∪ L 3 are exactly equal to the mathematical compositions of the state complexities of their component operations in the general cases. We also include a brief survey that summarizes all state complexity results for combined operations with two basic operations.

TCS Journal 2012 Journal Article

State complexity of union and intersection of square and reversal on k regular languages

• Yuan Gao
• Lila Kari
• Sheng Yu

In this paper, we continue our study on the state complexity of combined operations on regular languages. We study the state complexities of ⋃ i = 1 k L i 2, ⋂ i = 1 k L i 2, ⋃ i = 1 k L i R, and ⋂ i = 1 k L i R, for regular languages L i, 1 ≤ i ≤ k, k ≥ 2. We obtain the exact bounds for these combined operations and show that the state complexities of ⋃ i = 1 k L i 2 and ⋂ i = 1 k L i 2 are the same as the mathematical compositions of the state complexities of their component individual operations, while, on the other hand, the state complexities of ⋃ i = 1 k L i R and ⋂ i = 1 k L i R are lower than the corresponding mathematical compositions.

TCS Journal 2012 Journal Article

State complexity of union and intersection of star on regular languages

• Yuan Gao
• Lila Kari
• Sheng Yu

In this paper, we continue our study on state complexity of combined operations. We study the state complexities of L 1 ∗ ∪ L 2 ∗, ⋃ i = 1 k L i ∗, L 1 ∗ ∩ L 2 ∗, and ⋂ i = 1 k L i ∗ for regular languages L i, 1 ≤ i ≤ k. We obtain the exact bounds for these combined operations and show that the bounds are different from the mathematical compositions of the state complexities of their component individual operations.

TCS Journal 2009 Journal Article

Estimation of state complexity of combined operations

• Zoltán Ésik
• Yuan Gao
• Guangwu Liu
• Sheng Yu

It appears that the state complexity of each combined operation has its own special features. Thus, it is important and practical to obtain good estimates for some commonly used general cases. In this paper, we consider the state complexity of combined Boolean operations and give an exact bound for all of them in the case when the alphabet is not fixed. Moreover, we show that for any fixed alphabet, this bound can be reached in infinitely many cases. We also consider the state complexity of multiple catenations. The state complexities are obtained in the cases of the catenations of three and four languages. An estimate for the catenation of an arbitrary number of languages is given, which is very close to the state complexities in the three and four languages cases.

SODA Conference 2000 Conference Paper

Pattern discovery on character sets and real-valued data: linear bound on irredundant motifs and an efficient polynomial time algorithm

• Laxmi Parida
• Isidore Rigoutsos
• Aris Floratos
• Daniel E. Platt
• Yuan Gao