Author name cluster

Yu Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

117 papers

2 author rows

EAAI Journal 2026 Journal Article

A partial prior knowledge framework to discover the cycling visual environment coupling street view images and shared bike trajectory using deep learning

Qishen Guo
Shilei Hong
De Tong
Mengwei Xin
Yongxi Gong
Yu Liu

The effects of built environments on cycling have garnered considerable interests, and street view images (SVIs) as a proxy of built environments has been widely adopted to model cycling behaviours. However, most existing studies predominantly calculate indicators of built environment from segmented SVIs, and rely on prior knowledge, defined as pre-existing understandings of urban spaces, to verify well-known relationships or rules. This reliance limits the discovery of previously unrecognized environmental impacts on cycling. To address these challenges, we propose a partial prior knowledge framework for an explainable and accurate road-level cycling volume model that couples shared bike trajectory data and SVIs. This framework integrates deep learning and explainable machine learning to balance between prior-knowledge-based and end-to-end methods. Specifically, a Gradient-weighted Class Activation Mapping-enhanced deep learning method is applied to generate heatmaps of SVIs according to cycling volume data in an end-to-end mode, without any predefined assumptions about the street environment. Subsequently, semantic maps, as a typical form of prior knowledge, is incorporated to translate the highlighted regions into interpretable street visual elements with random forest regression and SHapley Additive exPlanations. The empirical study in Longgang District, Shenzhen demonstrates that our framework not only outperforms traditional approaches in prediction accuracy but also identifies several critical street visual elements and their impacts previously overlooked. The proposed framework improves human-artificial-intelligence interaction not only by enabling a deeper understanding of the complex impacts of street environments on cycling, but also by providing a generalization approach to extract human unknown knowledge from images and other non-conceptualized data.

Details DOI

AAAI Conference 2026 Conference Paper

Adaptive Dynamic Dehazing via Instruction-Driven and Task-Feedback Closed-Loop Optimization for Diverse Downstream Task Adaptation

Yafei Zhang
Shuaitian Song
Huafeng Li
Shujuan Wang
Yu Liu

In real-world vision systems, haze removal is required not only to enhance image visibility but also to meet the specific needs of diverse downstream tasks. To address this challenge, we propose a novel adaptive dynamic dehazing framework that incorporates a closed-loop optimization mechanism. It enables feedback-driven refinement based on downstream task performance and user instruction–guided adjustment during inference, allowing the model to satisfy the specific requirements of multiple downstream tasks without retraining. Technically, our framework integrates two complementary and innovative mechanisms: (1) a task feedback loop that dynamically modulates dehazing outputs based on performance across multiple downstream tasks, and (2) a text instruction interface that allows users to specify high-level task preferences. This dual-guidance strategy enables the model to adapt its dehazing behavior after training, tailoring outputs in real time to the evolving needs of multiple tasks. Extensive experiments across various vision tasks demonstrate the strong effectiveness, robustness, and generalizability of our approach. These results establish a new paradigm for interactive, task-adaptive dehazing that actively collaborates with downstream applications.

PDF Details DOI

EAAI Journal 2026 Journal Article

Causal-stabilized latent regression for robust industrial soft sensors

Kepeng Qiu
Baowei Rong
Yi Zhu
Yu Liu
Hui Yan
Haijun Cao
Weiwei Wang

Soft sensors play an essential role in monitoring and optimizing industrial processes. However, data redundancy and changing operating conditions often limit their reliability and ability to work well in different situations. This paper presents causal-stabilized latent regression (CSLR), a novel framework that combines causal inference with adaptive weighting. This combination helps balance feature importance and makes the model more stable when conditions change. The framework has two main parts working together: First, the causal balance weight optimization module reduces redundancy and aligns data distributions by optimizing both sample and feature weights through a causal weighting approach. Second, the robust weighted partial least squares (PLS) modeling module uses these weights to build a regression model that focuses on causally important features while reducing multicollinearity problems. Our analysis shows that CSLR effectively reduces feature redundancy and improves both generalization and stability when process conditions change. Tests on industrial debutanizer and fermentation processes demonstrate that CSLR achieves significant improvements over existing methods, with prediction error reduced by up to 43. 5%, R 2 increased by 8. 88%, and mean absolute percentage error decreased by 60. 62%, confirming its effectiveness for building accurate and reliable soft sensors.

Details DOI

AAAI Conference 2026 Conference Paper

Causality-Aligned Semantic Recovery for Incomplete Cross-Modal Retrieval

Haipeng Chen
Yu Liu
Xun Yang
Yuheng Liang
Yingda Lyu

Incomplete cross-modal retrieval (ICMR) requires models to recover missing modalities and robustly align heterogeneous ones for effective retrieval. Existing methods, however, fall short in both aspects. They often rely on limited semantic cues, such as single samples or coarse category prototypes, which compromises reconstruction quality. Moreover, these approaches are vulnerable to learning spurious cross-modal correlations, thereby impairing accurate alignment and hindering retrieval performance. To address these challenges, we propose Causality-Aligned Semantic Recovery (CASR), a novel method designed to both comprehensively restore missing modalities and mitigate spurious associations between vision and language. Our CASR involves two essential components: i) the Missing Modality Imagination (MMI) module, which combines category semantic priors with relevant contextual information to achieve high-quality semantic reconstruction; ii) the Explicit Causal Alignment (ECA) module, which explicitly learns environment-invariant attention, effectively eliminating the interference of spurious correlations and improving retrieval performance. Furthermore, we extend CASR to the challenging task of Partially Aligned Cross-Modal Retrieval, where we treat unlabeled unpaired data as a form of incomplete data. By leveraging MMI and ECA modules, we are able to learn robust representations in this setting. Extensive experiments on benchmark datasets under various missing rates demonstrate that CASR achieves superior robustness and retrieval performance.

PDF Details DOI

EAAI Journal 2026 Journal Article

Channel Clustering-based Attention Network for interpretable hard landing prediction

Hao Zhang
Huabo Sun
Yu Liu
Xinbin Zhao
Xu Li
Jiaxing Shang
Linjiang Zheng

Hard landing incidents are common flight safety events during the landing phase and are of significant concern in the aviation industry. Recent hard landing prediction methods tend to overemphasize temporal features while overlooking the landing process along the altitude dimension, importantly, altitude-based alignment enables more practical interpretability. Additionally, they often fail to capture dependencies between contributing factors and offer limited interpretability under fixed time windows. To address the above issues, we propose a Channel Clustering based Attention Network, termed CCAN, to predict hard landing incidents and identify their potential causes. Specifically, we resample and interpolate different flight parameters along the altitude dimension to align the landing process across different flights into a common reference frame for the subsequent interpretability. Subsequently, we design a channel clustering module that groups flight parameters into distinct clusters based on a predefined assignment threshold. Then, we employ graph attention network (GAT) to capture the dependencies between different flight parameters within and across clusters. To further reveal the interactions between flight parameters throughout the landing process, we incorporate attention mechanism into Gated Recurrent Units (GRUs) to extract informative temporal features. We conducted experiments on a real-world quick access recorders (QAR) dataset with 44, 729 Airbus A321 flights. Experimental results demonstrate that CCAN outperforms the baseline models in hard landing predictions and offers practical interpretability for hard landings by visualizing the dependencies between flight parameters and their interactions under the altitude reference during the landing process.

Details DOI

AAAI Conference 2026 Conference Paper

Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

Hao Tang
Yu Liu
Shuanglin Yan
Fei Shen
Shengfeng He
Jing Qin

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.

PDF Details DOI

JBHI Journal 2026 Journal Article

Depth-Induced Saliency Comparison Network for the Diagnosis of Alzheimer's Disease via Joint Analysis of Stimuli and Eye Movements

Yu Liu
Wenlin Zhang
Fangyu Zuo
Peiguang Jing
Yong Ji
Sun-Yuan Kung

Alzheimer's disease (AD) poses a growing global health challenge, with visuospatial impairments emerging as early indicators that can be detected through eye movement analysis. However, existing methods face two key limitations: first, they often analyze eye movements in isolation, without exploiting explicit saliency patterns for comparison; second, temporal attentional dynamics across sequences remain underexplored and fail to capture the temporal evolution of abnormal visuospatial patterns. To address these challenges, we propose a depth-induced saliency comparison network (DISCN). The DISCN first employs a depth-included salient attention module (DSAM) to construct comprehensive objective-subjective saliency priors from the saliency of RGB-D visual stimuli and normal control eye movements. A saliency-aware serial attention module (SSAM) then applies temporal attention to eye movements elicited by sequential stimuli to characterize dynamic visuospatial abnormalities. Experimental results on both internal and external cohorts demonstrate that the DISCN achieves robust performance in terms of distinguishing AD patients from normal controls by using eye movements.

Details DOI

AAAI Conference 2026 Conference Paper

Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning

Xinran Li
Yu Liu
Jiaqi Qiao
Xiujuan Xu

Emotion Recognition in Conversation (ERC) is a crucial task for understanding human emotions and enabling natural human-computer interaction. Although Large Language Models (LLMs) have recently shown great potential in this field, their ability to capture the intrinsic connections between explicit and implicit emotions remains limited. We propose a novel ERC training framework, PRC-Emo, which integrates Prompt engineering, demonstration Retrieval, and Curriculum learning, with the goal of exploring whether LLMs can effectively perceive emotions in conversational contexts. Specifically, we design emotion-sensitive prompt templates based on both explicit and implicit emotional cues to better guide the model in understanding the speaker’s psychological states. We construct the first dedicated demonstration retrieval repository for ERC, which includes training samples from widely used datasets, as well as high-quality dialogue examples generated by LLMs and manually verified. Moreover, we introduce a curriculum learning strategy into the LoRA fine-tuning process, incorporating weighted emotional shifts between same-speaker and different-speaker utterances to assign difficulty levels to dialogue samples, which are then organized in an easy-to-hard training sequence. Experimental results on two benchmark datasets—IEMOCAP and MELD—show that our method achieves new state-of-the-art (SOTA) performance, demonstrating the effectiveness and generalizability of our approach in improving LLM-based emotional understanding.

PDF Details DOI

AAAI Conference 2026 Conference Paper

EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision

Yifei Cao
Yu Liu
Guolong Wang
Zhu Liu
Kai Wang
Xianjie Zhang
Jizhe Yu
Xun Tu

Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-of-the-art performance on the Ego4D-VQ benchmark.

PDF Details DOI

AAAI Conference 2026 Conference Paper

FDP: A Frequency-Decomposition Preprocessing Pipeline for Unsupervised Anomaly Detection in Brain MRI

Hao Li
Zhenfeng Zhuang
Jingyu Lin
Yu Liu
Yifei Chen
Qiong Peng
Lequan Yu
Liansheng Wang

Due to the diversity of brain anatomy and the scarcity of annotated data, supervised anomaly detection for brain MRI remains challenging, driving the development of unsupervised anomaly detection (UAD) approaches. Current UAD methods typically utilize synthetically generated noise perturbations on healthy MRIs to train generative models for normal anatomy reconstruction, enabling anomaly detection via residual maps. However, such simulated anomalies lack the biophysical fidelity and morphological complexity characteristic of true clinical lesions. To advance UAD in brain MRI, we conduct the first systematic frequency-domain analysis of pathological signatures, revealing two key properties: (1) anomalies exhibit unique frequency patterns distinguishable from normal anatomy, and (2) low-frequency signals maintain consistent representations across healthy scans. These insights motivate our Frequency-Decomposition Preprocessing (FDP) framework—the first UAD method to leverage frequency-domain reconstruction for simultaneous pathology suppression and anatomical preservation. FDP can integrate seamlessly with existing anomaly simulation techniques, consistently enhancing detection performance across diverse architectures while maintaining diagnostic fidelity. Experimental results demonstrate that FDP consistently improves anomaly detection performance when integrated with existing methods. Notably, FDP achieves a 17.63% increase in DICE score with LDM while maintaining robust improvements across multiple baselines.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Gracefully Air-Written: Enhancing the Legibility and Style Consistency of In-Air Handwriting

Yu Liu
Cunrui Wang
Lin Feng
Jianxin Zhang
Bo Lu

Space computing devices expand handwritten input from two-dimensional screens into three-dimensional space, providing an unrestricted interactive experience. Due to the high degree of freedom and lack of tactile feedback in in-air handwriting, handwritten characters not only become less legible but also lose the writer's personal style. This paper proposes a method for reconstructing discrete in-air handwriting using continuous diffusion models, capturing the writing process and style from a small number of user-provided handwritten tracks and images, to restore the legibility of characters and mimics the writer's style. We represent handwritten track data in binary form and model it with continuous diffusion models, recovering discrete handwritten track data through threshold processing. Our approach reconstructs in-air handwritten characters in two stages. During the content preservation phase, we propose a partial noise injection strategy based on reference sequence modeling, using the content information of the original character as a guiding condition to maintain content consistency in handwritten character. In the style aggregation phase, we adaptively fuse the visual style of the handwritten in the image modality with the dynamic writing process in the sequence modality, overcoming issues of insufficient style capture due to noise interference in the backward process. Qualitative and quantitative experiments demonstrate the superiority of our method.

PDF Details DOI

AAAI Conference 2026 Conference Paper

IndoorUAV: Benchmarking Vision-Language UAV Navigation in Continuous Indoor Environments

Xu Liu
Yu Liu
Hanshuo Qiu
Yang Qirong
Zhouhui Lian

Vision-Language Navigation (VLN) enables agents to navigate in complex environments by following natural language instructions grounded in visual observations. Although most existing work has focused on ground-based robots or outdoor Unmanned Aerial Vehicles (UAVs), indoor UAV-based VLN remains underexplored, despite its relevance to real-world applications such as inspection, delivery, and search-and-rescue in confined spaces. To bridge this gap, we introduce IndoorUAV, a novel benchmark and method specifically tailored for VLN with indoor UAVs. We begin by curating over 1,000 diverse and structurally rich 3D indoor scenes from the Habitat simulator. Within these environments, we simulate realistic UAV flight dynamics to collect diverse 3D navigation trajectories manually, further enriched through data augmentation techniques. Furthermore, we design an automated annotation pipeline to generate natural language instructions of varying granularity for each trajectory. This process yields over 16,000 high-quality trajectories, comprising the IndoorUAV-VLN subset, which focuses on long-horizon VLN. To support short-horizon planning, we segment long trajectories into sub-trajectories by selecting semantically salient keyframes and regenerating concise instructions, forming the IndoorUAV-VLA subset. Finally, we introduce IndoorUAV-Agent, a novel navigation model designed for our benchmark, leveraging task decomposition and multimodal reasoning. We hope IndoorUAV serves as a valuable resource to advance research on vision-language embodied AI in the indoor aerial navigation domain.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Intention-Aware Diffusion Model for Pedestrian Trajectory Prediction

Yu Liu
Zhijie Liu
Xiao Ren
Youfu Li
He Kong

Predicting pedestrian motion trajectories is critical for the path planning and motion control of autonomous vehicles. Recent diffusion-based models have shown promising results in capturing the inherent stochasticity of pedestrian behavior for trajectory prediction. However, the absence of explicit semantic modelling of pedestrian intent in many diffusion-based methods may result in misinterpreted behaviors and reduced prediction accuracy. To address the above challenges, we propose a diffusion-based pedestrian trajectory prediction framework that incorporates both short-term and long-term motion intentions. Short-term intent is modelled using a residual polar representation, which decouples direction and magnitude to capture fine-grained local motion patterns. Long-term intent is estimated through a learnable, token-based endpoint predictor that generates multiple candidate goals with associated probabilities, enabling multimodal and context-aware intention modelling. Furthermore, we enhance the diffusion process by incorporating adaptive guidance and a residual noise predictor that dynamically refines denoising accuracy. The proposed framework is evaluated on the widely used ETH, UCY, NBA, and SDD benchmarks, demonstrating competitive results against state-of-the-art methods.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Learning 3D Occupancy from Beam Overlap in 2D Rotating mmWave Radar

Yu Du
Ruifeng Nie
Long Ma
Chengpei Xu
Yu Liu
Weimin Wang

Robust 3D perception under adverse weather is critical for autonomous systems. While mmWave Radars are inherently weather-resistant, conventional 2D rotating Radar sensors lack direct elevation resolution, limiting their 3D perception ability. Although 4D imaging radars can provide elevation information, they typically suffer from limited coverage and range. In this work, we exploit a key observation about mechanically rotating 2D mmWave Radars: in each sweep, an overlap exists between adjacent azimuth beam coverage due to the width of the main lobe, which makes the reflected intensity difference imply object materials and geometric shapes, including elevation. With this observation, we propose a method that learns 3D occupancy by disentangling bird’s-eye view (BEV) layout and elevation estimation from one frame Radar scan. Specifically, we partition one sweep into two interleaved subsets, corresponding to overlapping beam directions, and utilize them to infer coarse geometric structure through spatial differences and intensity patterns. Extensive quantitative and qualitative evaluations on two real-world datasets demonstrate that our proposed method outperforms existing baselines. The codes will be publicly available.

PDF Details DOI

AAAI Conference 2026 Conference Paper

MPMA: Preference Manipulation Attack Against Model Context Protocol

Zihan Wang
Rui Zhang
Yu Liu
Wenshu Fan
Wenbo Jiang
Qingchuan Zhao
Hongwei Li
Guowen Xu

Model Context Protocol (MCP) standardizes interface mapping for large language models (LLMs) to access external data and tools, which revolutionizes the paradigm of tool selection and facilitates the rapid expansion of the LLM agent tool ecosystem. However, as the MCP is increasingly adopted, third-party customized versions of the MCP server expose potential security vulnerabilities. In this paper, we first introduce a novel security threat, which we term the MCP Preference Manipulation Attack (MPMA). An attacker deploys a customized MCP server to manipulate LLMs, causing them to prioritize it over other competing MCP servers. This can result in economic benefits for attackers, such as revenue from paid MCP services or advertising income generated from free servers. To achieve MPMA, we first design a Direct Preference Manipulation Attack (DPMA) that achieves significant effectiveness by inserting the manipulative word and phrases into the tool name and description. However, such a direct modification is obvious to users and lacks stealthiness. To address these limitations, we further propose Genetic-based Advertising Preference Manipulation Attack (GAPMA). GAPMA employs four commonly used strategies to initialize descriptions and integrates a Genetic Algorithm (GA) to enhance stealthiness. The experiment results demonstrate that GAPMA balances high effectiveness and stealthiness. Our study reveals a critical vulnerability of the MCP in open ecosystems, highlighting an urgent need for robust defense mechanisms to ensure the fairness of the MCP ecosystem.

PDF Details DOI

AAAI Conference 2026 Conference Paper

OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval

Yu Liu
Yanbing Liu
Fangfang Yuan
Cong Cao
Youbang Sun
Kun Peng
WeiZhuo Chen
Jianjun Li

Recent advances in large language models (LLMs) and dense retrievers have driven significant progress in retrieval-augmented generation (RAG). However, existing approaches face significant challenges in complex reasoning-oriented multi-hop retrieval tasks: 1) Ineffective reasoning-oriented planning: Prior methods struggle to generate robust multi-step plans for complex queries, as rule-based decomposers perform poorly on out-of-template questions. 2) Suboptimal reasoning-driven retrieval: Related methods employ limited query reformulation, leading to iterative retrieval loops that often fail to locate golden documents. 3) Insufficient reasoning-guided filtering: Prevailing methods lack the fine-grained reasoning to effectively filter salient information from noisy results, hindering utilization of retrieved knowledge. Fundamentally, these limitations all stem from the weak coupling between retrieval and reasoning in current RAG architectures. We introduce the Orchestrated Planner-Executor Reasoning Architecture (OPERA), a novel reasoning-driven retrieval framework. OPERA's Goal Planning Module (GPM) decomposes questions into sub-goals, which are executed by a Reason-Execute Module (REM) with specialized components for precise reasoning and effective retrieval. To train OPERA, we propose Multi-Agents Progressive Group Relative Policy Optimization (MAPGRPO), a novel variant of GRPO. Experiments on complex multi-hop benchmarks show OPERA's superior performance, validating both the MAPGRPO method and OPERA's design.

PDF Details DOI

AAAI Conference 2026 Conference Paper

PathMind: A Retrieve-Prioritize-Reason Framework for Knowledge Graph Reasoning with Large Language Models

Yu Liu
Xixun Lin
Yanmin Shang
Yangxi Li
Shi Wang
Yanan Cao

Knowledge graph reasoning (KGR) is the task of inferring new knowledge by performing logical deductions on knowledge graphs. Recently, large language models (LLMs) have demonstrated remarkable performance in complex reasoning tasks. Despite promising success, current LLM-based KGR methods still face two critical limitations. First, existing methods often extract reasoning paths indiscriminately, without assessing their different importance, which may introduce irrelevant noise that misleads LLMs. Second, while some methods leverage LLMs to dynamically explore potential reasoning paths, they require high retrieval demands and frequent LLM calls. To address these limitations, we propose PathMind, a novel framework designed to enhance faithful and interpretable reasoning by selectively guiding LLMs with important reasoning paths. Specifically, PathMind follows a "Retrieve-Prioritize-Reason" paradigm. First, it retrieves a query subgraph from KG through the retrieval module. Next, it introduces a path prioritization mechanism that identifies important reasoning paths using a semantic-aware path priority function, which simultaneously considers the accumulative cost and the estimated future cost for reaching the target. Finally, PathMind generates accurate and logically consistent responses via a dual-phase training strategy, including task-specific instruction tuning and path-wise preference alignment. Extensive experiments on benchmark datasets demonstrate that PathMind consistently outperforms competitive baselines, particularly on complex reasoning tasks with fewer input tokens, by identifying essential reasoning paths.

PDF Details DOI

AAAI Conference 2026 Conference Paper

RSOD: Reliability-Guided Sonar Image Object Detection with Extremely Limited Labels

Chengzhou Li
Ping Guo
Guanchen Meng
Qi Jia
Jinyuan Liu
Zhu Liu
Xiaokang Liu
Yu Liu

Object detection in sonar images is a key technology in underwater detection systems. Compared to natural images, sonar images contain fewer texture details and are more susceptible to noise, making it difficult for non-experts to distinguish subtle differences between classes. This leads to their inability to provide precise annotation data for sonar images. Therefore, designing effective object detection methods for sonar images with extremely limited labels is particularly important. To address this, we propose a teacher-student framework called RSOD, which aims to fully learn the characteristics of sonar images and develop a pseudo-label strategy suitable for these images to mitigate the impact of limited labels. First, RSOD calculates a reliability score by assessing the consistency of the teacher's predictions across different views. To leverage this score, we introduce an object mixed pseudo-label method to tackle the shortage of labeled data in sonar images. Finally, we optimize the performance of the student by implementing a reliability-guided adaptive constraint. By taking full advantage of unlabeled data, the student can perform well even in situations with extremely limited labels. Notably, on the UATD dataset, our method, using only 5% of labeled data, achieves results that can compete against those of our baseline algorithm trained on 100% labeled data. We also collected a new dataset to provide more valuable data for research in the field of sonar.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Time Series Class-Incremental Learning via Confidence-guided Mask Distillation and Prototype-guided Contrastive Learning

Yu Liu
Haoqin Yang
Jinping Sui
Hui Wang
Haipeng Li
Weimin Wang
Qi Jia

Class-incremental learning (CIL) has recently gained great attention in the field of time series classification. Existing CIL methods based on knowledge distillation exhibit impressive ability to retain prior knowledge and overcome catastrophic forgetting, however, their effectiveness faces major challenges posed by time series data. Since temporal data is more susceptible to sensor errors and electronic noise, the distillation process may be significantly affected by noisy knowledge transfer. To address this issue, we propose a novel confidence-guided mask distillation (CMD) framework, to prevent the noisy inheritance during distillation. The core of CMD lies in a dynamic masking mechanism guided by prediction confidence, capable of allocating higher weights to high-confidence time series and substantially suppressing the influence of low-confidence ones. Additionally, different from prior work simply passing a set of feature prototypes to the classifier, we develop prototype-guided contrastive learning (PCL) to alleviate the classifier bias on new classes, through extra contrastive constraints to push away the feature distributions of old feature prototypes from those of new classes features. Extensive experiments on three time-series datasets demonstrate that, our method significantly outperforms other replay-free CIL approaches in raising average accuracy, as well as decreasing forgetting rate.

PDF Details DOI

EAAI Journal 2025 Journal Article

A hybrid deep learning and evolutionary framework for energy-aware interior augmentation via photorealistic visual illusions

Yu Liu
Haytham F. Isleem
Rupesh Kumar Tipu
Khalil El Hindi

Optical illusions can bias human judgments of size and brightness, yet their potential as a zero-hardware retrofit for reducing building energy use remains underexplored. We present a hybrid artificial intelligence framework that combines a diffusion-based generator with scene understanding and optimisation for energy-aware interior augmentation. The generator is adapted by low-rank adaptation (LoRA) of a Stable Diffusion version 2 backbone; indoor photographs are parsed by the Segment Anything Model (SAM) for semantic masks and a MiDaS monocular depth estimation model (MiDaS) for geometry; placement is solved by particle swarm optimisation (PSO) followed by proximal policy optimisation (PPO). An online reinforcement-learning adapter (RLA) enables rapid, scene-specific refinement. On a 30, 544-image corpus, the system attains a Fréchet Inception Distance (FID) of 18. 7, limits edge leakage to 1. 7%, and increases simulated magnitude-estimation scores of perceived size by 4. 9%. Based on an estimated brightness uplift, the required maintained illuminance can be reduced by 12% while preserving perceived spatial brightness — for example, from 300 lux (lx) to 264 lx — projecting to 9. 6% annual lighting-energy savings with light-emitting diode (LED) luminaires rated at 150 lumens per watt (lm/W). Implemented artificial intelligence components therefore include diffusion-based generative modelling, semantic segmentation, monocular depth estimation, evolutionary optimisation, and reinforcement learning, and the application is energy-aware interior design and lighting efficiency. Code, LoRA weights, and datasets are released to support replication and extension.

Details DOI

AAAI Conference 2025 Conference Paper

As Pseudo-Label Free as Possible: Leveraging Adaptive Feature Generation for Sparsely Annotated Object Detection

Shuilian Yao
Yu Liu
Qi Jia
Sihong Chen
Wei Zhuo

Compared to fully supervised object detection, training with sparse annotations typically leads to a decline in performance due to insufficient feature diversity. Existing sparsely annotated object detection (SAOD) methods often rely on pseudo-labeling strategies, but these pseudo-labels tend to introduce noise under extreme sparsity. To simultaneously avoid the impact of pseudo-label noise and enhance feature diversity, we propose a novel Adaptive Feature Generation (AdaptFG) model that generates features based on class names. This model integrates a pre-trained CLIP into a VAE-based feature generator, with its core innovation being an Adaptor that adaptively maps CLIP’s semantic embeddings to the object detector domain. Additionally, we introduce inter-class relationship reasoning in detector, which effectively mitigates misclassifications stemming from similar features. Extensive experimental results demonstrate that AdaptFG consistently outperforms state-of-the-art SAOD methods on the PASCAL VOC and MS COCO benchmarks.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

DoseSurv: Predicting Personalized Survival Outcomes under Continuous-Valued Treatments

Moritz Gögl
Yu Liu
Christopher Yau
Peter Watkinson
Tingting Zhu

Estimating heterogeneous treatment effects (HTEs) of continuous-valued interventions on survival, that is, time-to-event (TTE) outcomes, is crucial in various fields, notably in clinical decision-making and in driving the advancement of next-generation clinical trials. However, while HTE estimation for continuous-valued (i. e. , dosage-dependent) interventions and for TTE outcomes have been separately explored, their combined application remains largely overlooked in the machine learning literature. We propose DoseSurv, a varying-coefficient network designed to estimate HTEs for different dosage-dependent and non-dosage treatment options from TTE data. DoseSurv uses radial basis functions to model continuity in dose-response relationships and learns balanced representations to address covariate shifts arising in HTE estimation from observational TTE data. We present experiments across various treatment scenarios on both simulated and real-world data, demonstrating DoseSurv's superior performance over existing baseline models.

PDF Details

IJCAI Conference 2025 Conference Paper

EfficientPIE: Real-Time Prediction on Pedestrian Crossing Intention with Sole Observation

Fang Qu
Pengzhan Zhou
Yuepeng He
Kaixin Gao
Youyu Luo
Xin Feng
Yu Liu
Songtao Guo

Present Advanced Driving Assistance System (ADAS) responds to the dangerous crossing of pedestrians after the occurrence of the incident, occasionally causing severe accidents due to the stringent response window. Inference of pedestrian crossing intention may help vehicles operate in advance and enhance the safety of the vehicle by predicting the crossing probability. Recent studies usually ignore the demand of real-time forecast that required in the realistic driving scenario, and mainly focus on improving the model representation capacity on public datasets by increasing modality and observation time. Consequently, a new framework named EfficientPIE is proposed to predict the pedestrian crossing intention in real time with sole observation of the incident. To achieve reliable predictions, we propose incremental learning based on intention domain to relieve forgetting and promote performance with a progressive perturbation method. Our EfficientPIE outperforms all the SOTA models on two datasets PIE and JAAD, running nearly 7. 4x faster than the previously fastest model. Our code is available at https: //github. com/heinideyibadiaole/EfficientPIE.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

Enhancing Semantic Clarity: Discriminative and Fine-grained Information Mining for Remote Sensing Image-Text Retrieval

Yu Liu
Haipeng Chen
Yuheng Liang
Yuheng Yang
Xun Yang
Yingda Lyu

Remote sensing image-text retrieval is a fundamental task in remote sensing multimodal analysis, promoting the alignment of visual and language representations. The mainstream approaches commonly focus on capturing shared semantic representations between visual and textual modalities. However, the inherent characteristics of remote sensing image-text pairs lead to a semantic confusion problem, stemming from redundant visual representations and high inter-class similarity. To tackle this problem, we propose a novel Discriminative and Fine-grained Information Mining (DFIM) model, which aims to enhance semantic clarity by reducing visual redundancy and increasing the semantic gap between different classes. Specifically, the Dynamic Visual Enhancement (DVE) module adaptively enhances the visual discriminative features under the guidance of multimodal fusion information. Meanwhile, the Fine-grained Semantic Matching (FSM) module cleverly models the matching relationship between image regions and text words as an optimal transport problem, thereby refining intra-instance matching. Extensive experiments on two benchmark datasets justify the superiority of DFIM in terms of retrieval accuracy and visual interpretability over the leading methods.

PDF Details DOI

TMLR Journal 2025 Journal Article

Learning Deformable Body Interactions With Adaptive Spatial Tokenization

Hao Wang
Yu Liu
Daniel Biggs
Haoru Wang
Jiandong Yu
Ping Huang

Simulating interactions between deformable bodies is vital in fields like material science, mechanical design, and robotics. While learning-based methods with Graph Neural Networks (GNNs) are effective at solving complex physical systems, they encounter scalability issues when modeling deformable body interactions. To model interactions between objects, pairwise global edges have to be created dynamically, which is computationally intensive and impractical for large-scale meshes. To overcome these challenges, drawing on insights from geometric representations, we propose an Adaptive Spatial Tokenization (AST) method for efficient representation of physical states. By dividing the simulation space into a grid of cells and mapping unstructured meshes onto this structured grid, our approach naturally groups adjacent mesh nodes. We then apply a cross-attention module to map the sparse cells into a compact, fixed-length embedding, serving as tokens for the entire physical state. Self-attention modules are employed to predict the next state over these tokens in latent space. This framework leverages the efficiency of tokenization and the expressive power of attention mechanisms to achieve accurate and scalable simulation results. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches in modeling deformable body interactions. Notably, it remains effective on large-scale simulations with meshes exceeding 100,000 nodes, where existing methods are hindered by computational limitations. Additionally, we contribute a novel large-scale dataset encompassing a wide range of deformable body interactions to support future research in this area.

PDF Details

NeurIPS Conference 2025 Conference Paper

MiCo: Multi-image Contrast for Reinforcement Visual Reasoning

Xi Chen
Mingkang Zhu
Shaoteng Liu
Xiaoyang Wu
Xiaogang Xu
Yu Liu
Xiang Bai
Hengshuang Zhao

This work explores enabling Chain-of-Thought (CoT) reasoning to link visual cues across multiple images. A straightforward solution is to adapt rule-based reinforcement learning for Vision-Language Models (VLMs). However, such methods typically rely on manually curated question-answer pairs, which can be particularly challenging when dealing with fine-grained visual details and complex logic across images. Inspired by self-supervised visual representation learning, we observe that images contain inherent constraints that can serve as supervision. Based on this insight, we construct image triplets comprising two augmented views of the same image and a third, similar but distinct image. During training, the model is prompted to generate a reasoning process to compare these images (i. e. , determine same or different). Then we optimize the model with rule-based reinforcement learning. Due to the high visual similarity and the presence of augmentations, the model must attend to subtle visual cues and perform logical reasoning to succeed. Experimental results demonstrate that, although trained solely on visual comparison tasks, the learned reasoning ability generalizes effectively to a wide range of questions. Without relying on any human-annotated question-answer pairs, our method achieves significant improvements on multi-image reasoning benchmarks and shows strong performance on general vision tasks.

PDF Details

IJCAI Conference 2025 Conference Paper

MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving

Yichen Luo
Jia Wang
Dapeng Lan
Yu Liu
Zhibo Pang

Partial Differential Equations (PDEs) are fundamental for modeling physical systems, yet solving them in a generic and efficient manner using machine learning-based approaches remains challenging due to limited multi-input and multi-scale generalization capabilities, as well as high computational costs. This paper proposes the Multi-input and Multi-scale Efficient Transformer (MMET), a novel framework designed to address the above challenges. MMET decouples mesh and query points as two sequences and feeds them into the encoder and decoder, respectively, and uses a Gated Condition Embedding (GCE) layer to embed input variables or functions with varying dimensions, enabling effective solutions for multi-scale and multi-input problems. Additionally, a Hilbert curve-based reserialization and patch embedding mechanism decrease the input length. This significantly reduces the computational cost when dealing with large-scale geometric models. These innovations enable efficient representations and support multi-scale resolution queries for large-scale and multi-input PDE problems. Experimental evaluations on diverse benchmarks spanning different physical fields demonstrate that MMET outperforms SOTA methods in both accuracy and computational efficiency. This work highlights the potential of MMET as a robust and scalable solution for real-time PDE solving in engineering and physics-based applications, paving the way for future explorations into pre-trained large-scale models in specific domains. This work is open-sourced at https: //github. com/YichenLuo-0/MMET.

PDF Details DOI

TIST Journal 2025 Journal Article

Modeling N-ary Relational Knowledge Bases with Tensor Decomposition

Yu Liu
Quanming Yao
Yong Li

The binary relational knowledge base (KB, a.k.a. knowledge graph), representing real-world knowledge with binary relations and entities, has been an important research topic in artificial intelligence, while, considerable knowledge also involves beyond-binary relations. Recently, the area proposes to model n-ary relational KBs with both binary and beyond-binary relations included. However, most current models are extended from translational distance and neural network models in binary relational KBs, which suffer from weak expressiveness and high complexity, respectively. To overcome such issues, in this work, we propose a novel two-step modeling framework, GETD, generalizing the powerful tensor decomposition technique from binary relational KBs to the n-ary case. For n-ary relational KBs with single-arity relations, the GETD framework introduces Tucker decomposition and Tensor Ring decomposition for expressive and efficient modeling. Furthermore, the framework is technically extended for the representation of n-ary relational KBs with mixed-arity relations. The existing negative sampling technique is also generalized to the n-ary case for GETD. In addition, we theoretically prove that the GETD framework is fully expressive to completely represent any KBs. Empirical results on two representative datasets show that the proposed framework significantly outperforms the state-of-the-art methods, achieving 11–26% and 4–7% improvements on Hits@10 for the single-arity and the mixed-arity cases, respectively.

Details DOI

IROS Conference 2025 Conference Paper

Occlusion-Aware 6D Pose Estimation with Visual Observation Guided Diffusion Model

Yanbin Xiong
Buzhen Huang
Hui Ma
Yu Liu
Jun Cheng

Category-level 6D pose estimation in cluttered and occluded environments is a challenging task. Most existing methods rely on deterministic point-based correspondences to estimate target poses, which cannot consider the uncertainty for occluded objects, and thus result in inferior performance. In this paper, we propose a diffusion model guided by occlusion-aware observations to adaptively refine the object poses in occluded and cluttered scenes. Specifically, we first extract various 2D and 3D features from an RGB-D image to construct the conditions of diffusion model. In the reverse diffusion process, the model is guided by implicit correspondences, perception distance, and occlusion relationships to refine the noisy pose sampled from a standard Gaussian distribution. With several denoising steps, our method can produce accurate results that are consistent with image observations in occluded scenarios. The experimental results show that the proposed method can outperform baseline methods in major metrics in occlusion scenarios. Furthermore, our approach can also be applied in robotic grasping and manipulation tasks through grasping experiments in a cluttered enviroment on a physical UR5 robot.

Details

IJCAI Conference 2025 Conference Paper

OpenCarbon: A Contrastive Learning-based Cross-Modality Neural Approach for High-Resolution Carbon Emission Prediction Using Open Data

Jinwei Zeng
Yu Liu
Guozhen Zhang
Jingtao Ding
Yuming Lin
Jian Yuan
Yong Li

Accurately estimating high-resolution carbon emissions is crucial for effective emission governance and mitigation planning. While conventional methods for precise carbon accounting are hindered by substantial data collection efforts, the rise of open data and advanced learning techniques offers a promising solution. Once an open data-based prediction model is developed and trained, it can easily infer emissions for new areas based on available open data. To address this, we incorporate two modalities of open data, satellite images and point-of-interest (POI) data, to predict high-resolution urban carbon emissions, with satellite images providing macroscopic and static and POI data offering fine-grained and relatively dynamic functionality information. However, estimating high-resolution carbon emissions presents two significant challenges: the intertwined and implicit effects of various functionalities on carbon emissions, and the complex spatial contiguity correlations that give rise to the agglomeration effect. Our model, OpenCarbon, features two major designs that target the challenges: a cross-modality information extraction and fusion module to extract complementary functionality information from two modules and model their interactions, and a neighborhood-informed aggregation module to capture the spatial contiguity correlations. Extensive experiments demonstrate our model's superiority, with a significant performance gain of 26. 6% on R2. Further generalizability tests and case studies also show OpenCarbon's capacity to capture the intrinsic relation between urban functionalities and carbon emissions, validating its potential to empower efficient carbon governance and targeted carbon mitigation planning. Codes and data are available: https: //github. com/JinweiZzz/OpenCarbon.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection

Yu Liu
Hao Tang
Haiqi Zhang
Jing Qin
Zechao Li

Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications. While zero-shot OOD detection, which requires no training on in-distribution (ID) data, has become feasible with the emergence of vision-language models like CLIP, existing methods primarily focus on semantic matching and fail to fully capture distributional discrepancies. To address these limitations, we propose OT-DETECTOR, a novel framework that employs Optimal Transport (OT) to quantify both semantic and distributional discrepancies between test samples and ID labels. Specifically, we introduce cross-modal transport mass and transport cost as semantic-wise and distribution-wise OOD scores, respectively, enabling more robust detection of OOD samples. Additionally, we present a semantic-aware content refinement (SaCR) module, which utilizes semantic cues from ID labels to amplify the distributional discrepancy between ID and hard OOD samples. Extensive experiments on several benchmarks demonstrate that OT-DETECTOR achieves state-of-the-art performance across various OOD detection tasks, particularly in challenging hard-OOD scenarios.

PDF Details DOI

IROS Conference 2025 Conference Paper

SDF-guided Keyframe Selection: Novel Boost for NeRF SLAM Loop Closure

Hui Ma
Yu Liu
Jun Cheng

In the domain of Simultaneous Localization and Mapping (SLAM), loop closure is a linchpin for achieving accurate and consistent 3D environment mapping. However, the process is fraught with abrupt light changes and motion blur. These elements introduce uncertainties and inaccuracies in the data captured by sensors, severely undermining the system’s robustness. To address this critical challenge, we present a novel SDF-guided keyframe selection algorithm tailored for loop closure. Our approach capitalizes on the geometric insights provided by the Signed Distance Function (SDF) to meticulously choose keyframes, effectively mitigating the impact of noisy data. By doing so, we enhance the reliability of loop closure, refine the accuracy of 3D map reconstructions, and fortify the overall stability of the system. Our algorithm’s efficacy is substantiated through comprehensive experiments on datasets like Replica, ScanNet, and Tum-RGBD. Notably, it can be easily integrated as a plug-and-play module into diverse existing methods, enhancing their performance across different scenarios. Real-world trials using a hand-held LeTMC-520 camera for indoor scene reconstruction further validate its practicality and effectiveness.

Details

NeurIPS Conference 2025 Conference Paper

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Ruili Feng
Han Zhang
Zhilei Shu
Zhantao Yang
Longxiang Tang
Zhicai Wang
Andy Zheng
Jie Xiao

We present The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives. Trained on limited supervised data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains—deserts, grasslands, water bodies, and urban landscapes—in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting—an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

PDF Details

IROS Conference 2025 Conference Paper

ThermalLoc: A Vision Transformer-Based Approach for Robust Thermal Camera Relocalization in Large-Scale Environments

Yu Liu
Yangtao Meng
Xianfei Pan
Jie Jiang
Changhao Chen

Thermal cameras capture environmental data through heat emission, a fundamentally different mechanism compared to visible light cameras, which rely on pinhole imaging. As a result, traditional visual relocalization methods designed for visible light images are not directly applicable to thermal images. Despite significant advancements in deep learning for camera relocalization, approaches specifically tailored for thermal camera-based relocalization remain underexplored. To address this gap, we introduce ThermalLoc, a novel end-to-end deep learning method for thermal image relocalization. ThermalLoc effectively extracts both local and global features from thermal images by integrating EfficientNet with Transformers, and performs absolute pose regression using two MLP networks. We evaluated ThermalLoc on both the publicly available thermal-odometry dataset and our own dataset. The results demonstrate that ThermalLoc outperforms existing representative methods employed for thermal camera relocalization, including AtLoc, MapNet, PoseNet, and RobustLoc, achieving superior accuracy and robustness.

Details

NeurIPS Conference 2025 Conference Paper

TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster

Kanghui Ning
Zijie Pan
Yu Liu
Yushan Jiang
James Zhang
Kashif Rasul
Anderson Schneider
Lintao Ma

Large Language Models (LLMs) and Foundation Models (FMs) have recently become prevalent for time series forecasting tasks. While fine-tuning LLMs enables domain adaptation, they often struggle to generalize across diverse and unseen datasets. Moreover, existing Time Series Foundation Models (TSFMs) still face challenges in handling non-stationary dynamics and distribution shifts, largely due to the lack of effective mechanisms for adaptation. To this end, we present TS-RAG, a retrieval-augmented generation framework for time series forecasting that enhances the generalization and interpretability of TSFMs. Specifically, TS-RAG leverages pre-trained time series encoders to retrieve semantically relevant segments from a dedicated knowledge base, enriching the contextual representation of the input query. Furthermore, we propose an Adaptive Retrieval Mixer (ARM) module that dynamically fuses the retrieved patterns with the TSFM's internal representation, improving forecasting accuracy without requiring task-specific fine-tuning. Thorough empirical studies on seven public benchmark datasets demonstrate that TS-RAG achieves state-of-the-art zero-shot forecasting performance, outperforming the existing TSFMs by up to 6. 84\% across diverse domains while also providing desirable interpretability. Our code and data are available at: https: //github. com/UConn-DSIS/TS-RAG.

PDF Details

TMLR Journal 2025 Journal Article

UniZero: Generalized and Efficient Planning with Scalable Latent World Models

Yuan Pu
Yazhe Niu
Zhenjie Yang
Jiyuan Ren
Hongsheng Li
Yu Liu

Learning predictive world models is crucial for enhancing the planning capabilities of reinforcement learning (RL) agents. Recently, MuZero-style algorithms, leveraging the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in various domains. However, these methods struggle to scale in heterogeneous scenarios with diverse dependencies and task variability. To overcome these limitations, we introduce UniZero, a novel approach that employs a transformer-based world model to effectively learn a shared latent space. By concurrently predicting latent dynamics and decision-oriented quantities conditioned on the learned latent history, UniZero enables joint optimization of the long-horizon world model and policy, facilitating broader and more efficient planning in the latent space. We show that UniZero significantly outperforms existing baselines in benchmarks that require long-term memory. Additionally, UniZero demonstrates superior scalability in multitask learning experiments conducted on Atari benchmarks. In standard single-task RL settings, such as Atari and DMControl, UniZero matches or even surpasses the performance of current state-of-the-art methods. Finally, extensive ablation studies and visual analyses validate the effectiveness and scalability of UniZero's design choices. Our code is available at \textcolor{magenta}{https://github.com/opendilab/LightZero}.

PDF Details

NeurIPS Conference 2025 Conference Paper

ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models

Zixun Fang
Kai Zhu
Zhiheng Liu
Yu Liu
Wei Zhai
Yang Cao
Zheng-Jun Zha

Panoramic video generation aims to synthesize 360-degree immersive videos, holding significant importance in the fields of VR, world models, and spatial intelligence. Existing works fail to synthesize high-quality panoramic videos due to the inherent modality gap between panoramic data and perspective data, which constitutes the majority of the training data for modern diffusion models. In this paper, we propose a novel framework utilizing pretrained perspective video models for generating panoramic videos. Specifically, we design a novel panorama representation named ViewPoint map, which possesses global spatial continuity and fine-grained visual details simultaneously. With our proposed Pano-Perspective attention mechanism, the model benefits from pretrained perspective priors and captures the panoramic spatial correlations of the ViewPoint map effectively. Extensive experiments demonstrate that our method can synthesize highly dynamic and spatially consistent panoramic videos, achieving state-of-the-art performance and surpassing previous methods.

PDF Details

NeurIPS Conference 2025 Conference Paper

VividFace: A Robost and High-Fidelity Video Face Swapping Framework

Hao Shao
Shulun Wang
Yang Zhou
Guanglu Song
Dailan He
Zhuofan Zong
Shuo Qin
Yu Liu

Video face swapping has seen increasing adoption in diverse applications, yet existing methods primarily trained on static images struggle to address temporal consistency and complex real-world scenarios. To overcome these limitations, we propose the first video face swapping framework, VividFace, a robust and high-fidelity diffusion-based framework. VividFace employs a novel hybrid training strategy that leverages abundant static image data alongside temporal video sequences, enabling it to effectively model temporal coherence and identity consistency in videos. Central to our approach is a carefully designed diffusion model integrated with a specialized VAE, capable of processing image-video hybrid data efficiently. To further enhance identity and pose disentanglement, we introduce and release the Attribute-Identity Disentanglement Triplet (AIDT) dataset, comprising a large-scale collection of triplets where each set contains three face images—two sharing the same pose and two sharing the same identity. Augmented comprehensively with occlusion scenarios, AIDT significantly boosts the robustness of VividFace against occlusions. Moreover, we incorporate advanced 3D reconstruction techniques as conditioning inputs to address significant pose variations effectively. Extensive experiments demonstrate that VividFace achieves state-of-the-art performance in identity preservation, temporal consistency, and visual realism, surpassing existing methods while requiring fewer inference steps. Our framework notably mitigates common challenges such as temporal flickering, identity loss, and sensitivity to occlusions and pose variations. The AIDT dataset, source code, and pre-trained weights will be released to support future research. The code and pretrained weights are available on the project page.

PDF Details

NeurIPS Conference 2025 Conference Paper

Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance

Ruihang Chu
Yefei He
Zhekai Chen
Shiwei Zhang
Xiaogang Xu
Bin Xia
Dingdong WANG
Hongwei Yi

We present Wan-Move, a simple and scalable framework that brings motion control to video generative models. Existing motion-controllable methods typically suffer from coarse control granularity and limited scalability, leaving their outputs insufficient for practical use. We narrow this gap by achieving precise and high-quality motion control. Our core idea is to directly make the original condition features motion-aware for guiding video synthesis. To this end, we first represent object motions with dense point trajectories, allowing fine-grained control over the scene. We then project these trajectories into latent space and propagate the first frame's features along each trajectory, producing an aligned spatiotemporal feature map that tells how each scene element should move. This feature map serves as the updated latent condition, which is naturally integrated into the off-the-shelf image-to-video model, e. g. , Wan-I2V-14B, as motion guidance without any architecture change. It removes the need for auxiliary motion encoders and makes fine-tuning base models easily scalable. Through scaled training, Wan-Move generates 5-second, 480p videos whose motion controllability rivals Kling 1. 5 Pro's commercial Motion Brush, as indicated by user studies. To support comprehensive evaluation, we further design MoveBench, a rigorously curated benchmark featuring diverse content categories and hybrid-verified annotations. It is distinguished by larger data volume, longer video durations, and high-quality motion annotations. Extensive experiments on MoveBench and the public dataset consistently show Wan-Move's superior motion quality. Code, models, and benchmark data are made available.

PDF Details

YNIMG Journal 2025 Journal Article

White matter hyperintensity tissue property spatial variations as a function of cognitive status in Parkinson’s disease

Mariyemuguli Reheman
Sagar Buch
Naying He
Pei Huang
Qiurong Yu
Xinhui Wang
Yu Liu
Youmin Zhang

BACKGROUND AND PURPOSE: The pathological relationship between white matter hyperintensities (WMH) and cognitive impairment in Parkinson's disease (PD) remains unclear due to their variable locations, heterogeneity, and limited assessment of underlying tissue properties. This study integrates T2-FLAIR and quantitative MRI (qMRI) to investigate burden, spatial distribution, and extent of tissue alterations in WMH, aiming to elucidate their role in cognitive decline among PD patients. METHODS: A total of 122 age- and sex-matched PD patients and 65 healthy controls (HC) were recruited, with PD patients grouped by Montreal Cognitive Assessment (MoCA) score including normal, mild cognitive impairment (MCI) or PD with dementia (PDD). WMH burden was compared across groups and cognitive status. Water content, T1, and T2* measures were derived from qMRI data and tissue property heatmaps and periventricular distance profiles were constructed for all groups to visualize location-dependent tissue alterations of WMH relative to the lateral ventricles. In addition, voxel-wise analysis was performed to examine the correlation between WMH lesion tissue properties and MoCA scores. RESULTS: WMH volume was significantly higher in PDD compared to other groups (p < 0.05) and negatively correlated with MoCA scores (r = -0.352, p < 0.001). WMH appeared predominantly around the lateral ventricles, with anterior horn involvement common to all groups and posterior horn involvement specific to PDD. qMRI measures were significantly elevated in WMH compared to normal appearing white matter (NAWM) (p < 0.001), with heatmaps showing a negative gradient of tissue property changes from the lateral ventricles to the NAWM. Voxel-wise analysis revealed a significant negative correlation between the qMRI tissue properties of periventricular WMH and MoCA scores, with the strongest association observed in the periventricular WM situated just beyond the boundary of the lateral ventricles. CONCLUSION: Over and above volume differences, the spatial distribution and tissue property variations of WMH were closely linked to cognitive impairment in PD patients, with distinct patterns across different cognitive stages.

Details DOI

EAAI Journal 2024 Journal Article

A commercial vehicle weight prediction method based on driving simulation data

Yu Liu
Hao Zhang
Xianglei Zhu
Jingyuan Li
Hang Xu
Bo Zhang
Hanzhengnan Yu
Yu Wang

Neural network is beneficial for obtaining weight, which is a crucial parameter for vehicle in operation. However, the high demand of neural network for real data, and the lack of in-depth research on multi-vehicle weight prediction scenarios, have limited the development of weight prediction. In this paper, relying on the genetic algorithm-optimized BP neural network (GA-BP neural network), a commercial vehicle weight prediction method based on driving simulation data was proposed. The driving data of multiple commercial vehicle types were collected and used to study the weight prediction effect of the proposed method in single-vehicle and multi-vehicle scenarios. The fuzzy processing method of driving simulation data was proposed to process simulation data instead of real data to train the model, which effectively solves the problem of high demand for real data in neural network method. The study shows that proposed method can achieve weight prediction deviation of 2. 87% for single-vehicle model and 4. 63% for multi-vehicle model.

Details DOI

AAAI Conference 2024 Conference Paper

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

Yinmin Zhang
Jie Liu
Chuming Li
Yazhe Niu
Yaodong Yang
Yu Liu
Wanli Ouyang

Offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Built on offline RL algorithms, most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples. In this paper, from a novel perspective, we systematically study the challenges that remain in O2O RL and identify that the reason behind the slow improvement of the performance and the instability of online finetuning lies in the inaccurate Q-value estimation inherited from offline pretraining. Specifically, we demonstrate that the estimation bias and the inaccurate rank of Q-value cause a misleading signal for the policy update, making the standard offline RL algorithms, such as CQL and TD3-BC, ineffective in the online finetuning. Based on this observation, we address the problem of Q-value estimation by two techniques: (1) perturbed value update and (2) increased frequency of Q-value updates. The first technique smooths out biased Q-value estimation with sharp peaks, preventing early-stage policy exploitation of sub-optimal actions. The second one alleviates the estimation bias inherited from offline pretraining by accelerating learning. Extensive experiments on the MuJoco and Adroit environments demonstrate that the proposed method, named SO2, significantly alleviates Q-value estimation issues, and consistently improves the performance against the state-of-the-art methods by up to 83.1%.

PDF Details DOI

AAAI Conference 2024 Conference Paper

AUC Optimization from Multiple Unlabeled Datasets

Zheng Xie
Yu Liu
Ming Li

Weakly supervised learning aims to make machine learning more powerful when the perfect supervision is unavailable, and has attracted much attention from researchers. Among the various scenarios of weak supervision, one of the most challenging cases is learning from multiple unlabeled (U) datasets with only a little knowledge of the class priors, or U^m learning for short. In this paper, we study the problem of building an AUC (area under ROC curve) optimal model from multiple unlabeled datasets, which maximizes the pairwise ranking ability of the classifier. We propose U^m-AUC, an AUC optimization approach that converts the U^m data into a multi-label AUC optimization problem, and can be trained efficiently. We show that the proposed U^m-AUC is effective theoretically and empirically.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval

Yu Liu
Guihe Qin
Haipeng Chen
Zhiyong Cheng
Xun Yang

Text-based Person Retrieval (TPR) aims to retrieve relevant images of specific pedestrians based on the given textual query. The mainstream approaches primarily leverage pretrained deep neural networks to learn the mapping of visual and textual modalities into a common latent space for cross-modality matching. Despite their remarkable achievements, existing efforts mainly focus on learning the statistical cross-modality correlation found in training data, other than the intrinsic causal correlation. As a result, they often struggle to retrieve accurately in the face of environmental changes such as illumination, pose, and occlusion, or when encountering images with similar attributes. In this regard, we pioneer the observation of TPR from a causal view. Specifically, we assume that each image is composed of a mixture of causal factors (which are semantically consistent with text descriptions) and non-causal factors (retrieval-irrelevant, e.g., background), and only the former can lead to reliable retrieval judgments. Our goal is to extract text-critical robust visual representation (i.e., causal factors) and establish domain invariant cross-modality correlations for accurate and reliable retrieval. However, causal/non-causal factors are unobserved, so we emphasize that ideal causal factors that can simulate causal scenes should satisfy two basic principles:1） Independence: being independent of non-causal factors, and 2）Sufficiency: being causally sufficient for TPR across different environments. Building on that, we propose an Invariant Representation Learning method for TPR (IRLT), that enforces the visual representations to satisfy the two aforementioned critical properties. Extensive experiments on three datasets clearly demonstrate the advantages of IRLT over leading baselines in terms of accuracy and generalization.

PDF Details DOI

AAAI Conference 2024 Conference Paper

CI-STHPAN: Pre-trained Attention Network for Stock Selection with Channel-Independent Spatio-Temporal Hypergraph

Hongjie Xia
Huijie Ao
Long Li
Yu Liu
Sen Liu
Guangnan Ye
Hongfeng Chai

Quantitative stock selection is one of the most challenging FinTech tasks due to the non-stationary dynamics and complex market dependencies. Existing studies rely on channel mixing methods, exacerbating the issue of distribution shift in financial time series. Additionally, complex model structures they build make it difficult to handle very long sequences. Furthermore, most of them are based on predefined stock relationships thus making it difficult to capture the dynamic and highly volatile stock markets. To address the above issues, in this paper, we propose Channel-Independent based Spatio-Temporal Hypergraph Pre-trained Attention Networks (CI-STHPAN), a two-stage framework for stock selection, involving Transformer and HGAT based stock time series self-supervised pre-training and stock-ranking based downstream task fine-tuning. We calculate the similarity of stock time series of different channel in dynamic intervals based on Dynamic Time Warping (DTW), and further construct channel-independent stock dynamic hypergraph based on the similarity. Experiments with NASDAQ and NYSE markets data over five years show that our framework outperforms SOTA approaches in terms of investment return ratio (IRR) and Sharpe ratio (SR). Additionally, we find that even without introducing graph information, self-supervised learning based on the vanilla Transformer Encoder also surpasses SOTA results. Notable improvements are gained on the NYSE market. It is mainly attributed to the improvement of fine-tuning approach on Information Coefficient (IC) and Information Ratio based IC (ICIR), indicating that the fine-tuning method enhances the accuracy and stability of the model prediction.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

Dongzhi JIANG
Guanglu Song
Xiaoshi Wu
Renrui Zhang
Dazhong Shen
Zhuofan Zong
Yu Liu
Hongsheng Li

Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. We break down the problem into two causes: concept ignorance and concept mismapping. To tackle the two challenges, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with the image-to-text concept matching mechanism. Firstly, we introduce a novel image-to-text concept activation module to guide the diffusion model in revisiting ignored concepts. Additionally, an attribute concentration module is proposed to map the text conditions of each entity to its corresponding image area correctly. Extensive experimental evaluations, conducted across three distinct text-to-image alignment benchmarks, demonstrate the superior efficacy of our proposed method, CoMat-SDXL, over the baseline model, SDXL~\cite{podell2023sdxl}. We also show that our method enhances general condition utilization capability and generalizes to the long and complex prompt despite not specifically training on it.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Critic-Guided Decision Transformer for Offline Reinforcement Learning

Yuanfu Wang
Chao Yang
Ying Wen
Yu Liu
Yu Qiao

Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner. However, prevailing RCSL methods largely focus on deterministic trajectory modeling, disregarding stochastic state transitions and the diversity of future trajectory distributions. A fundamental challenge arises from the inconsistency between the sampled returns within individual trajectories and the expected returns across multiple trajectories. Fortunately, value-based methods offer a solution by leveraging a value function to approximate the expected returns, thereby addressing the inconsistency effectively. Building upon these insights, we propose a novel approach, termed the Critic-Guided Decision Transformer (CGDT), which combines the predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer. By incorporating a learned value function, known as the critic, CGDT ensures a direct alignment between the specified target returns and the expected returns of actions. This integration bridges the gap between the deterministic nature of RCSL and the probabilistic characteristics of value-based methods. Empirical evaluations on stochastic environments and D4RL benchmark datasets demonstrate the superiority of CGDT over traditional RCSL methods. These results highlight the potential of CGDT to advance the state of the art in offline RL and extend the applicability of RCSL to a wide range of RL tasks.

PDF Details DOI

JBHI Journal 2024 Journal Article

Deep Learning-Based Eye-Tracking Analysis for Diagnosis of Alzheimer's Disease Using 3D Comprehensive Visual Stimuli

Fangyu Zuo
Peiguang Jing
Jinglin Sun
Jizhong Duan
Yong Ji
Yu Liu

Alzheimer's Disease (AD) is a neurodegenerative disorder that causes a continuous decline in cognitive functions and eventually results in death. An early AD diagnosis is important for taking active measures to slow its deterioration. Traditional diagnoses are usually based on clinical experience, which is limited by several realistic factors. In this paper, we focus on exploiting deep learning techniques to diagnose AD based on eye-tracking behaviors. Visual attention, as a typical eye-tracking behavior, is of great clinical value in detecting cognitive abnormalities in AD patients. To better analyze the differences in visual attention between AD patients and normals, we first conducted a 3D comprehensive visual task on a noninvasive eye-tracking system to collect visual attention heatmaps. Then a multilayered comparison convolutional neural network (MC-CNN) is proposed to distinguish the visual attention differences between AD patients and normals. In MC-CNN, the multilayered feature representations of heatmaps were obtained by hierarchical residual blocks to better encode eye movement behaviors, which were further integrated into a distance vector to benefit the comprehensive visual task. From evaluation, MC-CNN can distinguish AD patients from normals with 0. 84 accuracy, 0. 86 recall, 0. 82 precision, 0. 83 F1-score and 0. 90 area under the curve (AUC). The above results demonstrate the effectiveness of the proposed MC-CNN in AD diagnosis based on the comprehensive 3D visual task.

Details DOI

AAMAS Conference 2024 Conference Paper

Dual-Policy-Guided Offline Reinforcement Learning with Optimal Stopping

Weibo Jiang
Shaohui Li
Zhi Li
Yuxin Ke
Zhizhuo Jiang
Yaowen Li
Yu Liu

Policy-guided offline reinforcement learning (POR) decomposes the offline reinforcement learning (offline RL) problem into goal estimation and goal-conditioned execution subproblems, leading to improved performance. However, we reveal that the preciseness of the estimated goal massively affects the performance and robustness of the trained goal-conditioned policy. To overcome this problem, we propose an offline RL model with dual guide-policies to improve the preciseness of the goal and reduce the variance. The proposed dual-policy-guided offline RL (Dual POR) adopts an integrating function, which balances the goals predicted by two guide-policies to obtain a refined goal. Moreover, we employ the optimal stopping strategy to schedule the training process, which dramatically shortens the training process and improves the generalization. The proposed Dual POR achieves state-of-the-art performance on the D4RL datasets with reduced variances. The improvements in highcomplexity tasks are even significant, which indicates the potential of the proposed Dual POR in real-world applications.

PDF

AAAI Conference 2024 Conference Paper

Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches

Yu Liu
Runzhe Wan
James McQueen
Doug Hains
Jinxiang Gu
Rui Song

The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency. Traditionally, experimenters determine AES based on domain knowledge. However, this method becomes impractical for online experimentation services managing numerous experiments, and a more automated approach is hence of great demand. We initiate the study of data-driven AES selection in for online experimentation services by introducing two solutions. The first employs a three-layer Gaussian Mixture Model considering the heteroskedasticity across experiments, and it seeks to estimate the true expected effect size among positive experiments. The second method, grounded in utility theory, aims to determine the optimal effect size by striking a balance between the experiment's cost and the precision of decision-making. Through comparisons with baseline methods using both simulated and real data, we showcase the superior performance of the proposed approaches.

PDF Details DOI

TMLR Journal 2024 Journal Article

Enhancing Vision-Language Model with Unmasked Token Alignment

Jihao Liu
Jinliang Zheng
Boxiao Liu
Yu Liu
Hongsheng Li

Contrastive pre-training on image-text pairs, exemplified by CLIP, becomes a standard technique for learning multi-modal visual-language representations. Although CLIP has demonstrated remarkable performance, training it from scratch on noisy web-scale datasets is computationally demanding. On the other hand, mask-then-predict pre-training approaches, like Masked Image Modeling (MIM), offer efficient self-supervised learning for single-modal representations. This paper introduces $\textbf{U}$nmasked $\textbf{T}$oken $\textbf{A}$lignment ($\textbf{UTA}$), a method that leverages existing CLIP models to further enhance its vision-language representations. UTA trains a Vision Transformer (ViT) by aligning unmasked visual tokens to the corresponding image tokens from a frozen CLIP vision encoder, which automatically aligns the ViT model with the CLIP text encoder. The pre-trained ViT can be directly applied for zero-shot evaluation even without training on image-text pairs. Compared to MIM approaches, UTA does not suffer from training-finetuning inconsistency and is much more training-efficient by avoiding using the extra $\mathrm{[MASK]}$ tokens. Extensive experimental results demonstrate that UTA can enhance CLIP models and outperform existing MIM methods on various uni- and multi-modal benchmarks.

PDF Details

AAAI Conference 2024 Conference Paper

Estimating On-Road Transportation Carbon Emissions from Open Data of Road Network and Origin-Destination Flow Data

Jinwei Zeng
Yu Liu
Jingtao Ding
Jian Yuan
Yong Li

Accounting for over 20% of the total carbon emissions, the precise estimation of on-road transportation carbon emissions is crucial for carbon emission monitoring and efficient mitigation policy formulation. However, existing estimation methods typically depend on hard-to-collect individual statistics of vehicle miles traveled to calculate emissions, thereby suffering from high data collection difficulty. To relieve this issue by utilizing the strong pattern recognition of artificial intelligence, we incorporate two sources of open data representative of the transportation demand and capacity factors, the origin-destination (OD) flow data and the road network data, to build a hierarchical heterogeneous graph learning method for on-road carbon emission estimation (HENCE). Specifically, a hierarchical graph consisting of the road network level, community level, and region level is constructed to model the multi-scale road network-based connectivity and travel connection between spatial areas. Heterogeneous graphs consisting of OD links and spatial links are further built at both the community level and region level to capture the intrinsic interactions between travel demand and road network accessibility. Extensive experiments on two large-scale real-world datasets demonstrate HENCE's effectiveness and superiority with R-squared exceeding 0.75 and outperforming baselines by 9.60% on average, validating its success in pioneering the use of artificial intelligence to empower carbon emission management and sustainability development. The implementation codes are available at this link: https://github.com/tsinghua-fib-lab/HENCE.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models

Bingqi Ma
Zhuofan Zong
Guanglu Song
Hongsheng Li
Yu Liu

Large language models based on decoder-only transformers have demonstrated superior text understanding capabilities compared to CLIP and T5-series models. However, the paradigm for utilizing current advanced LLMs in text-to-image diffusion models remains to be explored. We observed an unusual phenomenon: directly using a large language model as the prompt encoder significantly degrades the prompt-following ability in image generation. We identified two main obstacles behind this issue. One is the misalignment between the next token prediction training in LLM and the requirement for discriminative prompt features in diffusion models. The other is the intrinsic positional bias introduced by the decoder-only architecture. To deal with this issue, we propose a novel framework to fully harness the capabilities of LLMs. Through the carefully designed usage guidance, we effectively enhance the text representation capability of the LLM for prompt encoding and eliminate its inherent positional bias. This allows us to flexibly integrate state-of-the-art LLMs into the text-to-image generation model. Furthermore, we also provide an effective manner to fuse multiple LLMs into our framework. Considering the excellent performance and scaling capabilities demonstrated by the transformer architecture, we further design an LLM-Infused Diffusion Transformer (LI-DIT)based on the framework. We conduct extensive experiments to validate LI-DIT across model size and data size. Benefiting from the inherent ability of the LLMs and our innovative designs, the prompt understanding performance of LI-DIT easily surpasses state-of-the-art open-source models as well as mainstream closed-source commercial models including Stable Diffusion 3, DALL-E 3, and Midjourney V6.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

From Pixels to Progress: Generating Road Network from Satellite Imagery for Socioeconomic Insights in Impoverished Areas

Yanxin Xi
Yu Liu
Zhicheng Liu
Sasu Tarkoma
Pan Hui
Yong Li

The Sustainable Development Goals (SDGs) aim to resolve societal challenges, such as eradicating poverty and improving the lives of vulnerable populations in impoverished areas. Those areas rely on road infrastructure construction to promote accessibility and economic development. Although publicly available data like OpenStreetMap is available to monitor road status, data completeness in impoverished areas is limited. Meanwhile, the development of deep learning techniques and satellite imagery shows excellent potential for earth monitoring. To tackle the challenge of road network assessment in impoverished areas, we develop a systematic road extraction framework combining an encoder-decoder architecture and morphological operations on satellite imagery, offering an integrated workflow for interdisciplinary researchers. Extensive experiments of road network extraction on real-world data in impoverished regions achieve a 42. 7% enhancement in the F1-score over the baseline methods and reconstruct about 80% of the actual roads. We also propose a comprehensive road network dataset covering approximately 794, 178 km2 area and 17. 048 million people in 382 impoverished counties in China. The generated dataset is further utilized to conduct socioeconomic analysis in impoverished counties, showing that road network construction positively impacts regional economic development. The technical appendix, code, and generated dataset can be found at https: //github. com/tsinghua-fib-lab/Road_network_extraction_impoverished_counties.

PDF Details DOI

AAAI Conference 2024 Conference Paper

GMP-AR: Granularity Message Passing and Adaptive Reconciliation for Temporal Hierarchy Forecasting

Fan Zhou
Chen Pan
Lintao Ma
Yu Liu
Siqiao Xue
James Zhang
Jun Zhou
Hongyuan Mei

Time series forecasts of different temporal granularity are widely used in real-world applications, e.g., sales prediction in days and weeks for making different inventory plans. However, these tasks are usually solved separately without ensuring coherence, which is crucial for aligning downstream decisions. Previous works mainly focus on ensuring coherence with some straightforward methods, e.g., aggregation from the forecasts of fine granularity to the coarse ones, and allocation from the coarse granularity to the fine ones. These methods merely take the temporal hierarchical structure to maintain coherence without improving the forecasting accuracy. In this paper, we propose a novel granularity message-passing mechanism (GMP) that leverages temporal hierarchy information to improve forecasting performance and also utilizes an adaptive reconciliation (AR) strategy to maintain coherence without performance loss. Furthermore, we introduce an optimization module to achieve task-based targets while adhering to more real-world constraints. Experiments on real-world datasets demonstrate that our framework (GMP-AR) achieves superior performances on temporal hierarchical forecasting tasks compared to state-of-the-art methods. In addition, our framework has been successfully applied to a real-world task of payment traffic management in Alipay by integrating with the task-based optimization module.

PDF Details DOI

YNIMG Journal 2024 Journal Article

In Vivo visualization of white matter fiber tracts in the brainstem using low flip angle double echo 3D gradient echo imaging at 3T

Qiuyun Xu
Yongsheng Chen
Stephan Miller
Kunal Bajaj
Jairo Santana
Mohamed Badawy
Haiying Lyu
Yu Liu

BACKGROUND: White matter (WM) fiber tracts in the brainstem communicate with various regions in the cerebrum, cerebellum, and spinal cord. Clinically, small lesions, malformations, or histopathological changes in the brainstem can cause severe neurological disorders. A direct and non-invasive assessment approach could bring valuable information about the intricate anatomical variations of the white matter fiber tracts and nuclei. Although tractography from diffusion tensor imaging has been commonly used to map the WM fiber tracts connectivity, it is difficult to differentiate the complex WM tracts anatomically. Both high field MRI methods and ultrahigh-field MRI methods at 7T and 11.7 T have been used to enhance the contrast of WM fiber tracts. Despite their promising results, it is still challenging to achieve wide clinical adoption at 3T. In this study, we explored a clinically feasible method using a proton density weighted (PDW) 3D gradient echo (GRE) sequence to directly image the WM fiber tracts in the brainstem at 3T in vivo. METHODS: We optimized a 3D high resolution, double echo, short TR, PDW GRE sequence on 5 healthy volunteers using a clinical 3T scanner to visualize the complicated anatomy of WM fiber tracts in the brain stem. Tissue properties including T1, proton density and T2* from in vivo quantitative MRI data were used for simulations to determine the optimal flip angle for the sequence. The visualization of multiple WM fiber tracts in the brainstem was assessed qualitatively and quantitatively using relative contrast and contrast-to-noise ratio (CNR). To improve the CNR, the final images were created by averaging over all echoes from two consecutive scans at the optimal flip angle. The results were compared to anatomical atlases and histology sections to identify the major fiber tracts. All the identified major fiber tracts were labeled on axial, sagittal and coronal slices. RESULTS: The WM fiber tracts were found to have distinct hypointense signal throughout the brainstem and most of the major WM fiber tracts, such as the corticospinal tract, medial lemniscus, medial longitudinal fasciculus, and central tegmental tract, in the brainstem up to and including the thalamus were identified in all subjects. Both qualitative and quantitative evaluations showed that the 3° scan offered the best contrast for WM fiber tracts for a TR of 20 ms. The average over the first two echo times and two consecutive 3° scans gave a CNR of 47.8 ± 6.2 for the pyramidal tracts in particular and CNRs values greater than 6.5 ± 2.4 for the rest of the fiber tracts. CONCLUSIONS: All the major fiber tracts in the brainstem could be visualized. Given the reasonably short scan time of 10 min at 3T, double echo PDW GRE sequence is a very practical approach for clinical adoption.

Details DOI

NeurIPS Conference 2024 Conference Paper

Instruction-Guided Visual Masking

Jinliang Zheng
Jianxiong Li
Sijie Cheng
Yinan Zheng
Jiaming Li
Jihao Liu
Yu Liu
Jingjing Liu

Instruction following is crucial in contemporary LLM. However, when extended to multimodal setting, it often suffers from misalignment between specific textual instruction and targeted local region of an image. To achieve more accurate and nuanced multimodal instruction following, we introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model. By constructing visual masks for instruction-irrelevant regions, IVM-enhanced multimodal models can effectively focus on task-relevant image regions to better align with complex instructions. Specifically, we design a visual masking data generation pipeline and create an IVM-Mix-1M dataset with 1 million image-instruction pairs. We further introduce a new learning technique, Discriminator Weighted Supervised Learning (DWSL) for preferential IVM training that prioritizes high-quality data samples. Experimental results on generic multimodal tasks such as VQA and embodied robotic control demonstrate the versatility of IVM, which as a plug-and-play tool, significantly boosts the performance of diverse multimodal models, yielding new state-of-the-art results across challenging multimodal benchmarks. Code, model and data are available at https: //github. com/2toinf/IVM.

PDF Details DOI

TIST Journal 2024 Journal Article

KGDA: A Knowledge Graph Driven Decomposition Approach for Cellular Traffic Prediction

Jiahui Gong
Tong Li
Huandong Wang
Yu Liu
Xing Wang
Zhendong Wang
Chao Deng
Junlan Feng

Understanding and accurately predicting cellular traffic data is vital for communication operators and device users, as it facilitates efficient resource allocation and ensures superior service quality. However, large-scale cellular traffic data forecasting remains challenging due to intricate temporal variations and complex spatial relationships. This article proposes a Knowledge Graph Driven Decomposition Approach (KGDA) for precise cellular traffic prediction. The KGDA breaks down the impact of static environmental factors and dynamic autocorrelations of cellular traffic time series, enabling the capture of overall traffic changes and understanding of traffic dependence on past values. Specifically, we propose an urban knowledge graph to capture the static environmental context of base stations, mapping these entities into the same latent space while retaining static environmental knowledge. The cellular traffic is divided into a regular pattern and fluctuating residual components, with the KGDA comprising four modules: a Knowledge Graph Representation Learning model, a traffic regular pattern prediction module, a traffic residual dynamic prediction module, and an attentional fusion module. The first leverages graph neural networks to extract spatial contexts and predict regular patterns, the second utilizes the Bi-directional Long Short-Term Memory (Bi-LSTM) model to capture autocorrelations of traffic time series, and the final module integrates the patterns and residuals to produce the final prediction result. Comprehensive experiments demonstrate that our proposed model outperforms state-of-the-art models by more than 10% in forecasting cellular traffic.

Details DOI

NeurIPS Conference 2024 Conference Paper

LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment

Juelin Zhu
Shen Yan
Long Wang
Shengyue Zhang
Yu Liu
Maojun Zhang

We propose a new method named LoD-Loc for visual localization in the air. Unlike existing localization algorithms, LoD-Loc does not rely on complex 3D representations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given a coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypotheses to describe pose probability distribution and select a pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises a 6-DoF pose optimization algorithm to refine the previous result with a differentiable Gaussian-Newton method. As no public dataset exists for the studied problem, we collect two datasets with map levels of LoD3. 0 and LoD2. 0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. The code and dataset will be made available upon publication.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Long-term Detection and Monitory of Chinese Urban Village Using Satellite Imagery

Yuming Lin
Xin Zhang
Yu Liu
Zhenyu Han
Qingmin Liao
Yong Li

Urban villages are areas filled with rural-like improvised structures in Chinese cities, usually housing the most vulnerable groups. Under the guidance of the Sustainable Development Goals (SDGs), the Chinese government initiated renewal and redevelopment projects, underscoring the meticulous mapping and segmentation of urban villages. Satellite imagery is advanced and efficient in identifying urban villages and monitoring changes, but traditional methods neglect the morphological diversity in season, shape, size, spacing, and layout of urban villages, which is not satisfying for long-term wide-range data. Here, we design a targeted approach based on Tobler’s First Law of Geography, using curriculum labeling to solve morphological diversity and semi-automatically generate segmentation for urban village boundaries. Specifically, we use manually labeled data as seeds for pre-trained SegFormer models and incrementally fine-tune the model based on geographical proximity. The rigorous experimentation across five diverse cities substantiates the commendable efficacy of our methodology. IoU metric demonstrates a noteworthy improvement of over 119% to baseline. Our final results cover 265, 050 urban villages across 433 cities in China over the past 10 years, and the analysis reveals the uneven redevelopment by geography and city scale. We further examine the within-city distribution and verify the urban scaling law associated with several socio-economic factors. Our method can be used nationwide to decide redevelopment priority and resource tilt, contributing to SDG 11. 1 on affordable housing and upgrading slums. The code and dataset are available at https: //github. com/tsinghua-fib-lab/LtCUV.

PDF Details DOI

TMLR Journal 2024 Journal Article

MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learning

Jie Liu
Yinmin Zhang
Chuming Li
Zhiyuan You
Zhanhui Zhou
Chao Yang
Yaodong Yang
Yu Liu

Building a single generalist agent with strong zero-shot capability has recently sparked significant advancements. However, extending this capability to multi-agent decision making scenarios presents challenges. Most current works struggle with zero-shot transfer, due to two challenges particular to the multi-agent settings: (a) a mismatch between centralized training and decentralized execution; and (b) difficulties in creating generalizable representations across diverse tasks due to varying agent numbers and action spaces. To overcome these challenges, we propose a Mask-Based collaborative learning framework for Multi-Agent decision making (MaskMA). Firstly, we randomly mask part of the units and collaboratively learn the policies of unmasked units to handle the mismatch. In addition, MaskMA integrates a generalizable action representation by dividing the action space into intrinsic actions solely related to the unit itself and interactive actions involving interactions with other units. This flexibility allows MaskMA to tackle tasks with varying agent numbers and thus different action spaces. Extensive experiments in SMAC reveal MaskMA, with a single model trained on 11 training maps, can achieve an impressive 77.8% average zero-shot win rate on 60 unseen test maps by decentralized execution, while also performing effectively on other types of downstream tasks (e.g., varied policies collaboration, ally malfunction, and ad hoc team play).

PDF Details

NeurIPS Conference 2024 Conference Paper

MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Zhuofan Zong
Bingqi Ma
Dazhong Shen
Guanglu Song
Hao Shao
Dongzhi JIANG
Hongsheng Li
Yu Liu

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e. g. , the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

PDF Details DOI

AAMAS Conference 2024 Conference Paper

Mutual Information as Intrinsic Reward of Reinforcement Learning Agents for On-demand Ride Pooling

Xianjie Zhang
Jiahao Sun
Chen Gong
Kai Wang
Yifei Cao
Hao Chen
Yu Liu

The emergence of on-demand ride pooling services allows each vehicle to serve multiple passengers at a time, thus increasing drivers’ income and enabling passengers to travel at lower prices than taxi/car on-demand services. Although on-demand ride pooling services can bring so many benefits, ride pooling services need a well-defined matching strategy to maximize the benefits for all parties (passengers, drivers, aggregation companies and environment), especially the regional dispatching of vehicles has a significant impact on matching and revenue. Existing algorithms often only consider revenue maximization, which makes it difficult for requests with unusual distribution to get rides. How to increase revenue while ensuring a reasonable assignment of requests brings a challenge to ride pooling service companies (aggregation companies). In this paper, we propose a framework for vehicle dispatching for ride pooling tasks, which splits the city into discrete dispatching regions and uses the reinforcement learning (RL) algorithm to dispatch vehicles in these regions. We also consider the mutual information (MI) between vehicle and request distribution as the intrinsic reward of the RL algorithm to improve the correlation between their distributions, thus ensuring the possibility of getting a ride for unusually distributed requests. In experimental results on a real-world taxi dataset, we demonstrate that our framework can significantly increase revenue up to an average of 3% over the existing best on-demand ride pooling method. ∗Corresponding author This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024), N. Alechina, V. Dignum, M. Dastani, J. S. Sichman (eds.), May 6 – 10, 2024, Auckland, New Zealand. © 2024 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org).

PDF

NeurIPS Conference 2024 Conference Paper

Not Just Object, But State: Compositional Incremental Learning without Forgetting

Yanyi Zhang
Binglin Qiu
Qi Jia
Yu Liu
Ran He

Most incremental learners excessively prioritize object classes while neglecting various kinds of states (e. g. color and material) attached to the objects. As a result, they are limited in the ability to model state-object compositionality accurately. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), which enables the model to recognize a variety of state-object compositions in an incremental learning fashion. Since the lack of suitable datasets, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Composition Incremental Learner (CompILer), to overcome the ambiguous composition boundary. Specifically, we exploit multi-pool prompt learning, and ensure the inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting which injects object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer. Code and datasets are available at: https: //github. com/Yanyi-Zhang/CompILer.

PDF Details DOI

YNIMG Journal 2024 Journal Article

PENCIL imaging: A novel approach for neuromelanin sensitive MRI in Parkinson's disease

Peng Liu
Xinhui Wang
Youmin Zhang
Pei Huang
Zhijia Jin
Zenghui Cheng
Yongsheng Chen
Qiuyun Xu

BACKGROUND: Parkinson's disease (PD) is associated with the loss of neuromelanin (NM) and increased iron in the substantia nigra (SN). Magnetization transfer contrast (MTC) is widely used for NM visualization but has limitations in brain coverage and scan time. This study aimed to develop a new approach called Proton-density Enhanced Neuromelanin Contrast in Low flip angle gradient echo (PENCIL) imaging to visualize NM in the SN. METHODS: This study included 30 PD subjects and 50 healthy controls (HCs) scanned at 3T. PENCIL and MTC images were acquired. NM volume in the SN pars compacta (SNpc), normalized image contrast (Cnorm), and contrast-to-noise ratio (CNR) were calculated. The change of NM volume in the SNpc with age was analyzed using the HC data. A group analysis compared differences between PD subjects and HCs. Receiver operating characteristic (ROC) analysis and area under the curve (AUC) calculations were used to evaluate the diagnostic performance of NM volume and CNR in the SNpc. RESULTS: PENCIL provided similar visualization and structural information of NM compared to MTC. In HCs, PENCIL showed higher NM volume in the SNpc than MTC, but this difference was not observed in PD subjects. PENCIL had higher CNR, while MTC had higher Cnorm. Both methods revealed a similar pattern of NM volume in SNpc changes with age. There were no significant differences in AUCs between NM volume in SNpc measured by PENCIL and MTC. Both methods exhibited comparable diagnostic performance in this regard. CONCLUSIONS: PENCIL imaging provided improved CNR compared to MTC and showed similar diagnostic performance for differentiating PD subjects from HCs. The major advantage is PENCIL has rapid whole-brain coverage and, when using STAGE imaging, offers a one-stop quantitative assessment of tissue properties.

Details DOI

NeurIPS Conference 2024 Conference Paper

Phased Consistency Models

Fu-Yun Wang
Zhaoyang Huang
Alexander W. Bergman
Dazhong Shen
Peng Gao
Michael Lingelbach
Keqiang Sun
Weikang Bian

Consistency Models (CMs) have made significant progress in accelerating the generation of diffusion models. However, their application to high-resolution, text-conditioned image generation in the latent space remains unsatisfactory. In this paper, we identify three key flaws in the current design of Latent Consistency Models~(LCMs). We investigate the reasons behind these limitations and propose Phased Consistency Models (PCMs), which generalize the design space and address the identified limitations. Our evaluations demonstrate that PCMs outperform LCMs across 1--16 step generation settings. While PCMs are specifically designed for multi-step refinement, they achieve comparable 1-step generation results to previously state-of-the-art specifically designed 1-step methods. Furthermore, we show the methodology of PCMs is versatile and applicable to video generation, enabling us to train the state-of-the-art few-step text-to-video generator. Our code is available at https: //github. com/G-U-N/Phased-Consistency-Model.

PDF Details DOI

ICLR Conference 2024 Conference Paper

Space Group Constrained Crystal Generation

Rui Jiao
Wenbing Huang 0001
Yu Liu
Deli Zhao
Yang Liu 0005

Crystals are the foundation of numerous scientific and industrial applications. While various learning-based approaches have been proposed for crystal generation, existing methods neglect the spacegroup constraint which is crucial in describing the geometry of crystals and closely relevant to many desirable properties. However, considering spacegroup constraint is challenging owing to its diverse and nontrivial forms. In this paper, we reduce the spacegroup constraint into an equivalent formulation that is more tractable to be handcrafted into the generation process. In particular, we translate the spacegroup constraint into two cases: the basis constraint of the invariant exponential space of the lattice matrix and the Wyckoff position constraint of the fractional coordinates. Upon the derived constraints, we then propose DiffCSP++, a novel diffusion model that has enhanced a previous work DiffCSP by further taking spacegroup constraint into account. Experiments on several popular datasets verify the benefit of the involvement of the spacegroup constraint, and show that our DiffCSP++ achieves the best or comparable performance on crystal structure prediction and ab initio crystal generation.

Details

ICML Conference 2024 Conference Paper

StrokeNUWA - Tokenizing Strokes for Vector Graphic Synthesis

Zecheng Tang
Chenfei Wu
Zekai Zhang
Minheng Ni
Shengming Yin
Yu Liu
Zhengyuan Yang
Lijuan Wang

To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model’s ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation "stroke" tokens on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a $94\times$ speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6. 9%.

Details

ICLR Conference 2024 Conference Paper

The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models

Yan Liu 0002
Yu Liu
Xiaokang Chen
Pin-Yu Chen
Daoguang Zan
Min-Yen Kan
Tsung-Yi Ho

Pre-trained Language models (PLMs) have been acknowledged to contain harmful information, such as social biases, which may cause negative social impacts or even bring catastrophic results in application. Previous works on this problem mainly focused on using black-box methods such as probing to detect and quantify social biases in PLMs by observing model outputs. As a result, previous debiasing methods mainly finetune or even pre-train PLMs on newly constructed anti-stereotypical datasets, which are high-cost. In this work, we try to unveil the mystery of social bias inside language models by introducing the concept of {\sc Social Bias Neurons}. Specifically, we propose {\sc Integrated Gap Gradients (IG$^2$)} to accurately pinpoint units (i.e., neurons) in a language model that can be attributed to undesirable behavior, such as social bias. By formalizing undesirable behavior as a distributional property of language, we employ sentiment-bearing prompts to elicit classes of sensitive words (demographics) correlated with such sentiments. Our IG$^2$ thus attributes the uneven distribution for different demographics to specific Social Bias Neurons, which track the trail of unwanted behavior inside PLM units to achieve interoperability. Moreover, derived from our interpretable technique, {\sc Bias Neuron Suppression (BNS)} is further proposed to mitigate social biases. By studying BERT, RoBERTa, and their attributable differences from debiased FairBERTa, IG$^2$ allows us to locate and suppress identified neurons, and further mitigate undesired behaviors. As measured by prior metrics from StereoSet, our model achieves a higher degree of fairness while maintaining language modeling ability with low cost\footnote{This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups.}.

Details

AAAI Conference 2024 Conference Paper

UV-SAM: Adapting Segment Anything Model for Urban Village Identification

Xin Zhang
Yu Liu
Yuming Lin
Qingmin Liao
Yong Li

Urban villages, defined as informal residential areas in or around urban centers, are characterized by inadequate infrastructures and poor living conditions, closely related to the Sustainable Development Goals (SDGs) on poverty, adequate housing, and sustainable cities. Traditionally, governments heavily depend on field survey methods to monitor the urban villages, which however are time-consuming, labor-intensive, and possibly delayed. Thanks to widely available and timely updated satellite images, recent studies develop computer vision techniques to detect urban villages efficiently. However, existing studies either focus on simple urban village image classification or fail to provide accurate boundary information. To accurately identify urban village boundaries from satellite images, we harness the power of the vision foundation model and adapt the Segment Anything Model (SAM) to urban village segmentation, named UV-SAM. Specifically, UV-SAM first leverages a small-sized semantic segmentation model to produce mixed prompts for urban villages, including mask, bounding box, and image representations, which are then fed into SAM for fine-grained boundary identification. Extensive experimental results on two datasets in China demonstrate that UV-SAM outperforms existing baselines, and identification results over multiple years show that both the number and area of urban villages are decreasing over time, providing deeper insights into the development trends of urban villages and sheds light on the vision foundation models for sustainable cities. The dataset and codes of this study are available at https://github.com/tsinghua-fib-lab/UV-SAM.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Hao Shao
Shengju Qian
Han Xiao
Guanglu Song
Zhuofan Zong
Letian Wang
Yu Liu
Hongsheng Li

Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on this website to support further research in this area.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Zero-shot Image Editing with Reference Imitation

Xi Chen
Yutong Feng
Mengting Chen
Yiyang Wang
Shilong Zhang
Yu Liu
Yujun Shen
Hengshuang Zhao

Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e. g. , some relative pictures come across online), without having to cope with the fit between the reference and the source. Such a design requires the system to automatically figure out what to expect from the reference to perform the editing. For this purpose, we propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images in a self-supervised manner. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives. We also construct a benchmark to facilitate further research.

PDF Details DOI

AAAI Conference 2023 Conference Paper

ACE: Cooperative Multi-Agent Q-learning with Bidirectional Action-Dependency

Chuming Li
Jie Liu
Yinmin Zhang
Yuhong Wei
Yazhe Niu
Yaodong Yang
Yu Liu
Wanli Ouyang

Multi-agent reinforcement learning (MARL) suffers from the non-stationarity problem, which is the ever-changing targets at every iteration when multiple agents update their policies at the same time. Starting from first principle, in this paper, we manage to solve the non-stationarity problem by proposing bidirectional action-dependent Q-learning (ACE). Central to the development of ACE is the sequential decision making process wherein only one agent is allowed to take action at one time. Within this process, each agent maximizes its value function given the actions taken by the preceding agents at the inference stage. In the learning phase, each agent minimizes the TD error that is dependent on how the subsequent agents have reacted to their chosen action. Given the design of bidirectional dependency, ACE effectively turns a multi-agent MDP into a single-agent MDP. We implement the ACE framework by identifying the proper network representation to formulate the action dependency, so that the sequential decision process is computed implicitly in one forward pass. To validate ACE, we compare it with strong baselines on two MARL benchmarks. Empirical experiments demonstrate that ACE outperforms the state-of-the-art algorithms on Google Research Football and StarCraft Multi-Agent Challenge by a large margin. In particular, on SMAC tasks, ACE achieves 100% success rate on almost all the hard and super hard maps. We further study extensive research problems regarding ACE, including extension, generalization and practicability.

PDF Details DOI

JBHI Journal 2023 Journal Article

Bi-CapsNet: A Binary Capsule Network for EEG-Based Emotion Recognition

Yu Liu
Yi Wei
Chang Li
Juan Cheng
Rencheng Song
Xun Chen

In recent years, deep learning has gained widespread attention in electroencephalogram (EEG)-based emotion recognition. However, deep learning methods are usually time-consuming with a large amount of memory usage, which obstructs their practical usage on resource-constrained devices. In this paper, we propose a binary capsule network (Bi-CapsNet) for EEG emotion recognition with low computational cost and memory usage. The Bi-CapsNet binarizes 32-bit weights and activations to 1 b, and replaces floating-point operations with efficient bitwise operations. To address the issue of function discontinuity in backward propagation, we use a continuous function to approximate the binarization process. Two popular EEG emotion databases, namely, DEAP and DREAMER, are used for performance evaluation. In comparison to its full-precision counterpart, the Bi-CapsNet achieves a $>\! 25\times$ reduction on the computational cost and a $>\! 5\times$ reduction on the memory usage, while with only a $< $ 1% drop on the recognition accuracy. Compared to some state-of-the-art EEG emotion recognition methods, the proposed method obtains more competitive performance. In addition, the Bi-CapsNet is implemented on a mobile phone via an open-source binary inference framework named Bolt, and it achieves an $\sim\! 5\times$ inference acceleration in comparison to its full-precision counterpart.

Details DOI

AAMAS Conference 2023 Conference Paper

Centralized Cooperative Exploration Policy for Continuous Control Tasks

Chao Li
Chen Gong
Qiang He
Xinwen Hou
Yu Liu

Despite recent works making great progress in continuous control tasks, exploration in these tasks has remained insufficiently investigated. This paper proposes CCEP (Centralized Cooperative Exploration Policy), which utilizes estimation biases of value functions to contribute to the exploration capacity. CCEP keeps two value functions initialized with different parameters, and generates diverse policies with multiple exploration styles from a pair of value functions. In addition, a centralized policy framework ensures that CCEP achieves message delivery between multiple policies, furthermore contributing to exploring the environment cooperatively. Extensive experimental results demonstrate that CCEP achieves higher exploration capacity. Empirical analysis shows diverse exploration styles in the learned policies by CCEP, reaping benefits in more exploration regions. Besides, the exploration capabilities of CCEP have been demonstrated to outperform current state-ofthe-art methods on multiple continuous control tasks.

PDF

AAAI Conference 2023 Conference Paper

COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering

Mingrui Lao
Nan Pu
Yu Liu
Kai He
Erwin M. Bakker
Michael S. Lew

Audio-Visual Question Answering (AVQA) is a sophisticated QA task, which aims at answering textual questions over given video-audio pairs with comprehensive multimodal reasoning. Through detailed causal-graph analyses and careful inspections of their learning processes, we reveal that AVQA models are not only prone to over-exploit prevalent language bias, but also suffer from additional joint-modal biases caused by the shortcut relations between textual-auditory/visual co-occurrences and dominated answers. In this paper, we propose a COllabrative CAusal (COCA) Regularization to remedy this more challenging issue of data biases. Specifically, a novel Bias-centered Causal Regularization (BCR) is proposed to alleviate specific shortcut biases by intervening bias-irrelevant causal effects, and further introspect the predictions of AVQA models in counterfactual and factual scenarios. Based on the fact that the dominated bias impairing model robustness for different samples tends to be different, we introduce a Multi-shortcut Collaborative Debiasing (MCD) to measure how each sample suffers from different biases, and dynamically adjust their debiasing concentration to different shortcut correlations. Extensive experiments demonstrate the effectiveness as well as backbone-agnostic ability of our COCA strategy, and it achieves state-of-the-art performance on the large-scale MUSIC-AVQA dataset.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Customizable Image Synthesis with Multiple Subjects

Zhiheng Liu
Yifei Zhang
Yujun Shen
Kecheng Zheng
Kai Zhu
Ruili Feng
Yu Liu
Deli Zhao

Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, existing algorithms suffer from high training cost and low success rate along with increased number of subjects. Towards controllable image synthesis with multiple subjects as the constraints, this work studies how to efficiently represent a particular subject as well as how to appropriately compose different subjects. We find that the text embedding regarding the subject token already serves as a simple yet effective representation that supports arbitrary combinations without any model tuning. Through learning a residual on top of the base embedding, we manage to robustly shift the raw subject to the customized subject given various text conditions. We then propose to employ layout, a very abstract and easy-to-obtain prior, as the spatial guidance for subject arrangement. By rectifying the activations in the cross-attention map, the layout appoints and separates the location of different subjects in the image, significantly alleviating the interference across them. Using cross-attention map as the intermediary, we could strengthen the signal of target subjects and weaken the signal of irrelevant subjects within a certain region, significantly alleviating the interference across subjects. Both qualitative and quantitative experimental results demonstrate our superiority over state-of-the-art alternatives under a variety of settings for multi-subject customization.

PDF Details

YNIMG Journal 2023 Journal Article

Diagnosing Parkinson's disease by combining neuromelanin and iron imaging features using an automated midbrain template approach

Mojtaba Jokar
Zhijia Jin
Pei Huang
Ying Wang
Youmin Zhang
Yan Li
Zenghui Cheng
Yu Liu

BACKGROUND AND PURPOSE: Early diagnosis of Parkinson's disease (PD) is still a clinical challenge. Most previous studies using manual or semi-automated methods for segmenting the substantia nigra (SN) are time-consuming and, despite raters being well-trained, individual variation can be significant. In this study, we used a template-based, automatic, SN subregion segmentation pipeline to detect the neuromelanin (NM) and iron features in the SN and SN pars compacta (SNpc) derived from a single 3D magnetization transfer contrast (MTC) gradient echo (GRE) sequence in an attempt to develop a comprehensive imaging biomarker that could be used to diagnose PD. MATERIALS AND METHODS: volume, SNpc volume and iron content with a variety of thresholds as well as the N1 sign in diagnosing PD. Correlation analyses were performed to study the relationship between these imaging measures and the clinical scales in PD. RESULTS: = 0.04, p = 0.013) in PD patients. CONCLUSION: volume, SNpc volume and iron content) resulted in an AUC of 0.947 and provided a comprehensive set of imaging biomarkers that, potentially, could be used to diagnose PD clinically.

Details DOI

JBHI Journal 2023 Journal Article

Dual-Channel Neural Network for Atrial Fibrillation Detection From a Single Lead ECG Wave

Bo Fang
Junxin Chen
Yu Liu
Wei Wang
Ke Wang
Amit Kumar Singh
Zhihan Lv

With the dramatic progress of wearable devices, continuous collection of single lead ECG wave is able to be implemented in a comfortable fashion. Data mining on single lead ECG wave is therefore attracting increasing attention, where atrial fibrillation (AF) detection is a hot topic. In this paper, we propose a dual-channel neural network for AF detection from a single lead ECG wave. Two primary phases are included, the data preprocessing part followed by a dual-channel neural network. A two-stage denoising procedure is developed for data preprocessing, so as to tackle the high noise and disturbance which generally resides in the ECG wave collected by wearable devices. Then the time-frequency spectrum and Poincare plot of the denoised ECG signal are imported into the developed dual-channel neural network for feature extraction and AF detection. On the 2017 PhysioNet/CinC Challenge database, the F1 values were 0. 83, 0. 90, and 0. 75 for AF rhythm and normal rhythm, and other rhythm, respectively. The results well validate the effectiveness of the proposed method for AF detection from a single lead ECG wave, and also indicate its performance advantages over some state-of-the-art counterparts.

Details DOI

NeurIPS Conference 2023 Conference Paper

LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios

Yazhe Niu
Yuan Pu
Zhenjie Yang
Xueyan Li
Tong Zhou
Jiyuan Ren
Shuai Hu
Hongsheng Li

Building agents based on tree-search planning capabilities with learned models has achieved remarkable success in classic decision-making problems, such as Go and Atari. However, it has been deemed challenging or even infeasible to extend Monte Carlo Tree Search (MCTS) based algorithms to diverse real-world applications, especially when these environments involve complex action spaces and significant simulation costs, or inherent stochasticity. In this work, we introduce LightZero, the first unified benchmark for deploying MCTS/MuZero in general sequential decision scenarios. Specificially, we summarize the most critical challenges in designing a general MCTS-style decision-making solver, then decompose the tightly-coupled algorithm and system design of tree-search RL methods into distinct sub-modules. By incorporating more appropriate exploration and optimization strategies, we can significantly enhance these sub-modules and construct powerful LightZero agents to tackle tasks across a wide range of domains, such as board games, Atari, MuJoCo, MiniGrid and GoBigger. Detailed benchmark results reveal the significant potential of such methods in building scalable and efficient decision intelligence. The code is available as part of OpenDILab at https: //github. com/opendilab/LightZero.

PDF Details

YNICL Journal 2023 Journal Article

Locus coeruleus and substantia nigra neuromelanin magnetic resonance imaging differentiates Parkinson’s disease and essential tremor

Xinhui Wang
Pei Huang
Ewart Mark Haacke
Yu Liu
Youmin Zhang
Zhijia Jin
Yan Li
Qiuyun Xu

BACKGROUND: Differential diagnosis of essential tremor (ET) and Parkinson's disease (PD) can still be a challenge in clinical practice. These two tremor disorders may have different pathogenesis related to the substantia nigra (SN) and locus coeruleus (LC). Characterizing neuromelanin (NM) in these structures may help improve the differential diagnosis. METHODS: from ET was assessed with a receiver operative characteristic curve, and the area under the curve (AUC) was calculated. RESULTS: from ET. CONCLUSION: and ET, and the investigation of the underlying pathophysiology.

Details DOI

NeurIPS Conference 2023 Conference Paper

RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths

Zeyue Xue
Guanglu Song
Qiushan Guo
Boxiao Liu
Zhuofan Zong
Yu Liu
Ping Luo

Text-to-image generation has recently witnessed remarkable achievements. We introduce a text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by stacking tens of mixture-of-experts (MoEs) layers, i. e. , space-MoE and time-MoE layers, enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting a particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2. 0, DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. Secondly, a single model with three billion parameters, trained on 1, 000 A100 GPUs for two months, achieves a state-of-the-art zero-shot FID score of 6. 61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers of image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https: //raphael-painter. github. io/.

PDF Details

AAAI Conference 2023 Conference Paper

SLOTH: Structured Learning and Task-Based Optimization for Time Series Forecasting on Hierarchies

Fan Zhou
Chen Pan
Lintao Ma
Yu Liu
Shiyu Wang
James Zhang
Xinxin Zhu
Xuanwei Hu

Multivariate time series forecasting with hierarchical structure is widely used in real-world applications, e.g., sales predictions for the geographical hierarchy formed by cities, states, and countries. The hierarchical time series (HTS) forecasting includes two sub-tasks, i.e., forecasting and reconciliation. In the previous works, hierarchical information is only integrated in the reconciliation step to maintain coherency, but not in forecasting step for accuracy improvement. In this paper, we propose two novel tree-based feature integration mechanisms, i.e., top-down convolution and bottom-up attention to leverage the information of the hierarchical structure to improve the forecasting performance. Moreover, unlike most previous reconciliation methods which either rely on strong assumptions or focus on coherent constraints only, we utilize deep neural optimization networks, which not only achieve coherency without any assumptions, but also allow more flexible and realistic constraints to achieve task-based targets, e.g., lower under-estimation penalty and meaningful decision-making loss to facilitate the subsequent downstream tasks. Experiments on real-world datasets demonstrate that our tree-based feature integration mechanism achieves superior performances on hierarchical forecasting tasks compared to the state-of-the-art methods, and our neural optimization networks can be applied to real-world tasks effectively without any additional effort under coherence and task-based constraints.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Style-Content Metric Learning for Multidomain Remote Sensing Object Recognition

Wenda Zhao
Ruikai Yang
Yu Liu
You He

Previous remote sensing recognition approaches predominantly perform well on the training-testing dataset. However, due to large style discrepancies not only among multidomain datasets but also within a single domain, they suffer from obvious performance degradation when applied to unseen domains. In this paper, we propose a style-content metric learning framework to address the generalizable remote sensing object recognition issue. Specifically, we firstly design an inter-class dispersion metric to encourage the model to make decision based on content rather than the style, which is achieved by dispersing predictions generated from the contents of both positive sample and negative sample and the style of input image. Secondly, we propose an intra-class compactness metric to force the model to be less style-biased by compacting classifier's predictions from the content of input image and the styles of positive sample and negative sample. Lastly, we design an intra-class interaction metric to improve model's recognition accuracy by pulling in classifier's predictions obtained from the input image and positive sample. Extensive experiments on four datasets show that our style-content metric learning achieves superior generalization performance against the state-of-the-art competitors. Code and model are available at: https://github.com/wdzhao123/TSCM.

PDF Details DOI

PRL Workshop 2023 Workshop Paper

Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning

Chuming Li
Ruonan Jia
Jiawei Yao
Jie Liu
Yinmin Zhang
Yazhe Niu
Yaodong Yang
Yu Liu

Model-based reinforcement learning (RL) has demonstrated remarkable successes on a range of continuous control tasks due to its high sample efficiency. To save the computation cost of conducting planning online, recent practices tend to distill optimized action sequences into an RL policy during the training phase. Although the distillation can incorporate both the foresight of planning and the exploration ability of RL policies, the theoretical understanding of these methods is yet unclear. In this paper, we extend the policy improvement step of Soft Actor-Critic (SAC) by developing an approach to distill from model-based planning to the policy. We then demonstrate that such an approach of policy improvement has a theoretical guarantee of monotonic improvement and convergence to the maximum value defined in SAC. We discuss effective design choices and implement our theory as a practical algorithm---$\textit{\textbf{M}odel-based \textbf{P}lanning \textbf{D}istilled to \textbf{P}olicy (MPDP)}$---that updates the policy jointly over multiple future time steps. Extensive experiments show that MPDP achieves better sample efficiency and asymptotic performance than both model-free and model-based planning algorithms on six continuous control benchmark tasks in MuJoCo.

PDF Details

TIST Journal 2023 Journal Article

UrbanKG: An Urban Knowledge Graph System

Yu Liu
Jingtao Ding
Yanjie Fu
Yong Li

Every day, our living city produces a tremendous amount of spatial-temporal data, involved with multiple sources from the individual scale to the city scale. Undoubtedly, such massive urban data can be explored for a better city and better life, as what the urban computing community has been dedicating in recent years. Nevertheless, existing studies are still facing the challenges of data fusion for the urban data as well as the knowledge distillation for specific applications. Moreover, there is a lack of full-featured and user-friendly platforms for both researchers and developers in the urban computing scenario. Therefore, in this article, we present UrbanKG, an urban knowledge graph system to incorporate a knowledge graph with urban computing. Specifically, the system introduces a complete scheme to construct a knowledge graph for urban data fusion. Built upon the data layer, the system further develops the multiple layers of construction, storage, algorithm, operation, and applications, which achieve knowledge distillation and support various functions to the users. We perform representative use cases and demonstrate the system capability of boosting performance in various downstream applications, indicating a promising research direction for knowledge-driven urban computing.

Details DOI

IJCAI Conference 2023 Conference Paper

Video Diffusion Models with Local-Global Context Guidance

Siyuan Yang
Lu Zhang
Yu Liu
Zhizhuo Jiang
You He

Diffusion models have emerged as a powerful paradigm in video synthesis tasks including prediction, generation, and interpolation. Due to the limitation of the computational budget, existing methods usually implement conditional diffusion models with an autoregressive inference pipeline, in which the future fragment is predicted based on the distribution of adjacent past frames. However, only the conditions from a few previous frames can't capture the global temporal coherence, leading to inconsistent or even outrageous results in long-term video prediction. In this paper, we propose a Local-Global Context guided Video Diffusion model (LGC-VD) to capture multi-perception conditions for producing high-quality videos in both conditional/unconditional settings. In LGC-VD, the UNet is implemented with stacked residual blocks with self-attention units, avoiding the undesirable computational cost in 3D Conv. We construct a local-global context guidance strategy to capture the multi-perceptual embedding of the past fragment to boost the consistency of future prediction. Furthermore, we propose a two-stage training strategy to alleviate the effect of noisy frames for more stable predictions. Our experiments demonstrate that the proposed method achieves favorable performance on video prediction, interpolation, and unconditional video generation. We release code at https: //github. com/exisas/LGC-VD.

PDF Details DOI

AAAI Conference 2022 System Paper

A Trend-Driven Fashion Design System for Rapid Response Marketing in E-commerce

Lianghua Huang
Yu Liu
Bin Wang
Pan Pan
Rong Jin

Fashion is the way we express ourselves and has grown into one of the largest industries in the world. Despite the significant evolvement of the fashion industry over the past decades, it is still a great challenge to respond to the diverse preferences of a large number of different consumers in time and accurately. To deal with the problem, we present an innovative demonstration of a trend-driven fashion design system using deep generative modeling, which enables automatic fashion design and editing based on trend reports. Our system consists of three components, including trend-driven fashion design, interactive fashion editing, and popularity estimation. The system offers a unified framework for mass-production of fashion designs that conform to the trend, which helps businesses better respond to market demands.

PDF Details

JBHI Journal 2022 Journal Article

DDCNN: A Deep Learning Model for AF Detection From a Single-Lead Short ECG Signal

Zhaocheng Yu
Junxin Chen
Yu Liu
Yongyong Chen
Tingting Wang
Robert Nowak
Zhihan Lv

With the popularity of the wireless body sensor network, real-time and continuous collection of single-lead electrocardiogram (ECG) data becomes possible in a convenient way. Data mining from the collected single-lead ECG waves has therefore aroused extensive attention worldwide, where early detection of atrial fibrillation (AF) is a hot research topic. In this paper, a two-channel convolutional neural network combined with a data augmentation method is proposed to detect AF from single-lead short ECG recordings. It consists of three modules, the first module denoises the raw ECG signals and produces 9-s ECG signals and heart rate (HR) values. Then, the ECG signals and HR rate values are fed into the convolutional layers for feature extraction, followed by three fully connected layers to perform the classification. The data augmentation method is used to generate synthetic signals to enlarge the training set and increase the diversity of the single-lead ECG signals. Validation experiments and the comparison with state-of-the-art studies demonstrate the effectiveness and advantages of the proposed method.

Details DOI

NeurIPS Conference 2022 Conference Paper

Large-batch Optimization for Dense Visual Predictions: Training Faster R-CNN in 4.2 Minutes

Zeyue Xue
Jianming Liang
Guanglu Song
Zhuofan Zong
Liang Chen
Yu Liu
Ping Luo

Training a large-scale deep neural network in a large-scale dataset is challenging and time-consuming. The recent breakthrough of large-batch optimization is a promising way to tackle this challenge. However, although the current advanced algorithms such as LARS and LAMB succeed in classification models, the complicated pipelines of dense visual predictions such as object detection and segmentation still suffer from the heavy performance drop in the large-batch training regime. To address this challenge, we propose a simple yet effective algorithm, named Adaptive Gradient Variance Modulator (AGVM), which can train dense visual predictors with very large batch size, enabling several benefits more appealing than prior arts. Firstly, AGVM can align the gradient variances between different modules in the dense visual predictors, such as backbone, feature pyramid network (FPN), detection, and segmentation heads. We show that training with a large batch size can fail with the gradient variances misaligned among them, which is a phenomenon primarily overlooked in previous work. Secondly, AGVM is a plug-and-play module that generalizes well to many different architectures (e. g. , CNNs and Transformers) and different tasks (e. g. , object detection, instance segmentation, semantic segmentation, and panoptic segmentation). It is also compatible with different optimizers (e. g. , SGD and AdamW). Thirdly, a theoretical analysis of AGVM is provided. Extensive experiments on the COCO and ADE20K datasets demonstrate the superiority of AGVM. For example, AGVM demonstrates more stable generalization performance than prior arts under extremely large batch size (i. e. , 10k). AGVM can train Faster R-CNN+ResNet50 in 4. 2 minutes without losing performance. It enables training an object detector with one billion parameters in just 3. 5 hours, reducing the training time by 20. 9×, whilst achieving 62. 2 mAP on COCO. The deliverables will be released at https: //github. com/Sense-X/AGVM.

PDF Details

IJCAI Conference 2022 Conference Paper

Memory Augmented State Space Model for Time Series Forecasting

Yinbo Sun
Lintao Ma
Yu Liu
Shijun Wang
James Zhang
Yangfei Zheng
Hu Yun
Lei Lei

State space model (SSM) provides a general and flexible forecasting framework for time series. Conventional SSM with fixed-order Markovian assumption often falls short in handling the long-range temporal dependencies and/or highly non-linear correlation in time-series data, which is crucial for accurate forecasting. To this extend, we present External Memory Augmented State Space Model (EMSSM) within the sequential Monte Carlo (SMC) framework. Unlike the common fixed-order Markovian SSM, our model features an external memory system, in which we store informative latent state experience, whereby to create ``memoryful" latent dynamics modeling complex long-term dependencies. Moreover, conditional normalizing flows are incorporated in our emission model, enabling the adaptation to a broad class of underlying data distributions. We further propose a Monte Carlo Objective that employs an efficient variational proposal distribution, which fuses the filtering and the dynamic prior information, to approximate the posterior state with proper particles. Our results demonstrate the competitiveness of forecasting performance of our proposed model comparing with other state-of-the-art SSMs.

PDF Details DOI

YNIMG Journal 2022 Journal Article

Plasticity changes in dorsolateral prefrontal cortex associated with procedural sequence learning are hemisphere-specific

Na Cao
Yanling Pi
Fanghui Qiu
Yanqiu Wang
Xue Xia
Yu Liu
Jian Zhang

Corticocortical neuroplastic changes from higher-order cortices to primary motor cortex (M1) have been described for procedural sequence learning. The dorsolateral prefrontal cortex (DLPFC) plays critical roles in cognition, including in motor learning and memory. However, neuroplastic changes in the DLPFC and their influence on M1 and on motor learning are not well understood. The present study examined bilateral DLPFC-M1 changes in plasticity induced by procedural motor sequence learning in a serial reaction time task. DLPFC plasticity induced by procedural sequence learning was examined by comparing before vs. after training assessments of ipsilateral/contralateral DLPFC-M1 interactions between sequence order and random order trials performed using either the left or right hand. Intra-hemispheric (inter-stimulus interval [ISI] = 10 ms) and inter-hemispheric (ISI = 10 or 50 ms) DLPFC-M1 interactions and single-pulse motor-evoked potentials (MEPs) were measured with transcranial magnetic stimulation (TMS). The reaction times of participants measured during motor training were faster for sequence learning than for random learning with either hand. Paired-pulse TMS induced DLPFC-M1 interactions that were disinhibited after motor sequence learning, especially for left DLPFC-left M1 interactions with right hand task performance and for left DLPFC-right M1 interactions with left hand task performance. These findings indicate that motor sequence learning induces neuroplastic changes to enhance DLPFC-M1 interactions. This manifestation of plasticity showed hemispheric specificity, favoring the left DLPFC. DLPFC plasticity may be a useful index of DLPFC function and may be a treatment target for enhancing DLPFC function and motor learning.

Details DOI

YNIMG Journal 2022 Journal Article

Subthalamic and pallidal stimulation in Parkinson's disease induce distinct brain topological reconstruction

Chunguang Chu
Naying He
Kristina Zeljic
Zhen Zhang
Jiang Wang
Jun Li
Yu Liu
Youmin Zhang

The subthalamic nucleus (STN) and globus pallidus internus (GPi) are the two most common and effective target brain areas for deep brain stimulation (DBS) treatment of advanced Parkinson's disease. Although DBS has been shown to restore functional neural circuits of this disorder, the changes in topological organization associated with active DBS of each target remain unknown. To investigate this, we acquired resting-state functional magnetic resonance imaging (fMRI) data from 34 medication-free patients with Parkinson's disease that had DBS electrodes implanted in either the subthalamic nucleus or internal globus pallidus (n = 17 each), in both ON and OFF DBS states. Sixteen age-matched healthy individuals were used as a control group. We evaluated the regional information processing capacity and transmission efficiency of brain networks with and without stimulation, and recorded how stimulation restructured the brain network topology of patients with Parkinson's disease. For both targets, the variation of local efficiency in motor brain regions was significantly correlated (p < 0.05) with improvement rate of the Uniform Parkinson's Disease Rating Scale-III scores, with comparable improvements in motor function for the two targets. However, non-motor brain regions showed changes in topological organization during active stimulation that were target-specific. Namely, targeting the STN decreased the information transmission of association, limbic and paralimbic regions, including the inferior frontal gyrus angle, insula, temporal pole, superior occipital gyri, and posterior cingulate, as evidenced by the simultaneous decrease of clustering coefficient and local efficiency. GPi-DBS had a similar effect on the caudate and lenticular nuclei, but enhanced information transmission in the cingulate gyrus. These effects were not present in the DBS-OFF state for GPi-DBS, but persisted for STN-DBS. Our results demonstrate that DBS to the STN and GPi induce distinct brain network topology reconstruction patterns, providing innovative theoretical evidence for deciphering the mechanism through which DBS affects disparate targets in the human brain.

Details DOI

JBHI Journal 2021 Journal Article

Emotion Recognition From Multi-Channel EEG via Deep Forest

Juan Cheng
Meiyao Chen
Chang Li
Yu Liu
Rencheng Song
Aiping Liu
Xun Chen

Recently, deep neural networks (DNNs) have been applied to emotion recognition tasks based on electroencephalography (EEG), and have achieved better performance than traditional algorithms. However, DNNs still have the disadvantages of too many hyperparameters and lots of training data. To overcome these shortcomings, in this article, we propose a method for multi-channel EEG-based emotion recognition using deep forest. First, we consider the effect of baseline signal to preprocess the raw artifact-eliminated EEG signal with baseline removal. Secondly, we construct 2$D$ frame sequences by taking the spatial position relationship across channels into account. Finally, 2$D$ frame sequences are input into the classification model constructed by deep forest that can mine the spatial and temporal information of EEG signals to classify EEG emotions. The proposed method can eliminate the need for feature extraction in traditional methods and the classification model is insensitive to hyperparameter settings, which greatly reduce the complexity of emotion recognition. To verify the feasibility of the proposed model, experiments were conducted on two public DEAP and DREAMER databases. On the DEAP database, the average accuracies reach to 97. 69% and 97. 53% for valence and arousal, respectively; on the DREAMER database, the average accuracies reach to 89. 03%, 90. 41%, and 89. 89% for valence, arousal and dominance, respectively. These results show that the proposed method exhibits higher accuracy than the state-of-art methods.

Details DOI

IJCAI Conference 2021 Conference Paper

Neighborhood Intervention Consistency: Measuring Confidence for Knowledge Graph Link Prediction

Kai Wang
Yu Liu
Quan Z. Sheng

Link prediction based on knowledge graph embeddings (KGE) has recently drawn a considerable momentum. However, existing KGE models suffer from insufficient accuracy and hardly evaluate the confidence probability of each predicted triple. To fill this critical gap, we propose a novel confidence measurement method based on causal intervention, called Neighborhood Intervention Consistency (NIC). Unlike previous confidence measurement methods that focus on the optimal score in a prediction, NIC actively intervenes in the input entity vector to measure the robustness of the prediction result. The experimental results on ten popular KGE models show that our NIC method can effectively estimate the confidence score of each predicted triple. The top 10% triples with high NIC confidence can achieve 30% higher accuracy in the state-of-the-art KGE models.

PDF Details DOI

JBHI Journal 2021 Journal Article

PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography

Rencheng Song
Huan Chen
Juan Cheng
Chang Li
Yu Liu
Xun Chen

Remote photoplethysmography (rPPG) is a non-contact technique for measuring cardiac signals from facial videos. High-quality rPPG pulse signals are urgently demanded in many fields, such as health monitoring and emotion recognition. However, most of the existing rPPG methods can only be used to get average heart rate (HR) values due to the limitation of inaccurate pulse signals. In this paper, a new framework based on generative adversarial network, called PulseGAN, is introduced to generate realistic rPPG pulse signals through denoising the chrominance (CHROM) signals. Considering that the cardiac signal is quasi-periodic and has apparent time-frequency characteristics, the error losses defined in time and spectrum domains are both employed with the adversarial loss to enforce the model generating accurate pulse waveforms as its reference. The proposed framework is tested on three public databases. The results show that the PulseGAN framework can effectively improve the waveform quality, thereby enhancing the accuracy of HR, the interbeat interval (IBI) and the related heart rate variability (HRV) features. The proposed method significantly improves the quality of waveforms compared to the input CHROM signals, with the mean absolute error of AVNN (the average of all normal-to-normal intervals) reduced by 41. 19%, 40. 45%, 41. 63%, and the mean absolute error of SDNN (the standard deviation of all NN intervals) reduced by 37. 53%, 44. 29%, 58. 41%, in the cross-database test on the UBFC-RPPG, PURE, and MAHNOB-HCI databases, respectively. This framework can be easily integrated with other existing rPPG methods to further improve the quality of waveforms, thereby obtaining more reliable IBI features and extending the application scope of rPPG techniques.

Details DOI

AAAI Conference 2021 Conference Paper

Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning

Yu Liu
Lianghua Huang
Pan Pan
Bin Wang
Yinghui Xu
Rong Jin

This paper presents a simple unsupervised visual representation learning method with a pretext task of discriminating all images in a dataset using a parametric, instance-level classifier. The overall framework is a replica of a supervised classification model, where semantic classes (e. g. , dog, bird, and ship) are replaced by instance IDs. However, scaling up the classification task from thousands of semantic labels to millions of instance labels brings specific challenges including 1) the large-scale softmax computation; 2) the slow convergence due to the infrequent visiting of instance samples; and 3) the massive number of negative classes that can be noisy. This work presents several novel techniques to handle these difficulties. First, we introduce a hybrid parallel training framework to make large-scale training feasible. Second, we present a raw-feature initialization mechanism for classification weights, which we assume offers a contrastive prior for instance discrimination and can clearly speed up converge in our experiments. Finally, we propose to smooth the labels of a few hardest classes to avoid optimizing over very similar negative pairs. While being conceptually simple, our framework achieves competitive or superior performance compared to state-of-the-art unsupervised approaches, i. e. , SimCLR, Mo- CoV2, and PIC under ImageNet linear evaluation protocol and on several downstream visual tasks, verifying that full instance classification is a strong pretraining technique for many semantic visual tasks.

PDF Details

IS Journal 2020 Journal Article

Joint Intelligence Ranking by Federated Multiplicative Update

Chi Zhang
Yu Liu
Le Wang
Yuehu Liu
Li Li
Nanning Zheng

The joint intelligence ranking of intelligent systems like autonomous driving is of great importance for building a more general, extensive, and universally accepted intelligence evaluation scheme. However, due to issues such as privacy security and industry or area competition, the integration of isolated test results may face large unimaginable difficulty in information security and encrypted model training. To address this, we derive the federated multiplicative update (FMU) algorithm with boundary constraints to solve the nonnegative matrix factorization based joint intelligence ranking. The encrypted learning process is developed to alternate original computation steps in multiplicative update algorithms. Owning feasible property for the fast convergence and secure exchange of variables, the proposed framework outperforms the previous work on both real and simulated data. Further experimental analysis reveals that the introduced federated mechanism does not harm the overall time efficiency.

Details DOI

AAAI Conference 2020 Conference Paper

KPNet: Towards Minimal Face Detector

Guanglu Song
Yu Liu
Yuhang Zang
Xiaogang Wang
Biao Leng
Qingsheng Yuan

The small receptive ﬁeld and capacity of minimal neural networks limit their performance when using them to be the backbone of detectors. In this work, we ﬁnd that the appearance feature of a generic face is discriminative enough for a tiny and shallow neural network to verify from the background. And the essential barriers behind us are 1) the vague deﬁnition of the face bounding box and 2) tricky design of anchor-boxes or receptive ﬁeld. Unlike most topdown methods for joint face detection and alignment, the proposed KPNet detects small facial keypoints instead of the whole face by in the bottom-up manner. It ﬁrst predicts the facial landmarks from a low-resolution image via the welldesigned ﬁne-grained scale approximation and scale adaptive soft-argmax operator. Finally, the precise face bounding boxes, no matter how we deﬁne it, can be inferred from the keypoints. Without any complex head architecture or meticulous network designing, the KPNet achieves state-of-theart accuracy on generic face detection and alignment benchmarks with only ∼ 1M parameters, which runs at 1000fps on GPU and is easy to perform real-time on most modern frontend chips.

PDF Details

IJCAI Conference 2020 Conference Paper

Label-Attended Hashing for Multi-Label Image Retrieval

Yanzhao Xie
Yu Liu
Yangtao Wang
Lianli Gao
Peng Wang
Ke Zhou

For the multi-label image retrieval, the existing hashing algorithms neglect the dependency between objects and thus fail to capture the attention information in the feature extraction, which affects the precision of hash codes. To address this problem, we explore the inter-dependency between objects through their co-occurrence correlation from the label set and adopt Multi-modal Factorized Bilinear (MFB) pooling component so that the image representation learning can capture this attention information. We propose a Label-Attended Hashing (LAH) algorithm which enables an end-to-end hash model with inter-dependency feature extraction. LAH first combines Convolutional Neural Network (CNN) and Graph Convolution Network (GCN) to separately generate the image representation and label co-occurrence embeddings, then adopts MFB to fuse these two modal vectors, finally learns the hash function with a Cauchy distribution based loss function via back propagation. Extensive experiments on public multi-label datasets demonstrate that (1) LAH can achieve the state-of-the-art retrieval results and (2) the usage of co-occurrence relationship and MFB not only promotes the precision of hash codes but also accelerates the hash learning. GitHub address: https: //github. com/IDSM-AI/LAH.

PDF Details DOI

YNIMG Journal 2020 Journal Article

Optimizing neuromelanin contrast in the substantia nigra and locus coeruleus using a magnetization transfer contrast prepared 3D gradient recalled echo sequence

Yu Liu
Junchen Li
Naying He
Yongsheng Chen
Zhijia Jin
Fuhua Yan
E. Mark Haacke

Neuromelanin (NM) loss in the substantia nigra (SN) and locus coeruleus (LC) is being investigated as an imaging biomarker for Parkinson’s disease (PD) using magnetization transfer contrast (MTC) magnetic resonance imaging. The MTC pulse operates in a way to suppress tissue with high macromolecular content thereby highlighting the presence of NM in the LC and the SN. The MTC pulse also leads to a reduction in the effective T1 of the tissue. In the past, a 3D gradient echo (GRE) sequence has usually been run with a single flip angle (FA) generally to highlight the T1 shortening effect when trying to visualize NM. We contend that the NM will be best seen with a low FA (relative to the Ernst angle) because the NM has high water content relative to the surrounding tissues. Therefore, the goal of this paper was to optimize the NM contrast in the SN and LC as a function of flip angle using a 3D GRE MTC strategically acquired gradient echo (STAGE) imaging approach. In order to accomplish this, short repeat time (62 ms), 3D GRE imaging data were collected for 7 different flip angles ranging from 5° to 40° for 14 healthy volunteers (age range 24–43 years, mean ± SD = 34. 8 ± 6. 0 years, 6 males). By measuring the contrast-to-noise ratio between these structures and the surrounding tissues, we found that the FA showing the best NM contrast was 15° - 20° for the SN and 20° - 25° for the LC. Using STAGE imaging with just two flip angles (15° and 30°) made it possible to quantify not only tissue properties such as T1 and proton density but also to generate synthetic MTC images at an arbitrary FA. These synthetic images make it possible to optimize the contrast for any changes in tissue property that might occur in the LC or SN as a function of age or disease. In conclusion, practically, two scans could be collected in roughly 7 min each for both FAs in a standard clinical imaging setting to evaluate the signal intensity and volume of the NM in the LC and SN.

Details DOI

JMLR Journal 2020 Journal Article

Smoothed Nonparametric Derivative Estimation using Weighted Difference Quotients

Yu Liu
Kris De Brabanter

Derivatives play an important role in bandwidth selection methods (e.g., plug-ins), data analysis and bias-corrected confidence intervals. Therefore, obtaining accurate derivative information is crucial. Although many derivative estimation methods exist, the majority require a fixed design assumption. In this paper, we propose an effective and fully data-driven framework to estimate the first and second order derivative in random design. We establish the asymptotic properties of the proposed derivative estimator, and also propose a fast selection method for the tuning parameters. The performance and flexibility of the method is illustrated via an extensive simulation study. [abs] [ pdf ][ bib ] &copy JMLR 2020. ( edit, beta )

PDF Details

AAAI Conference 2020 Conference Paper

Temporal Interlacing Network

Hao Shao
Shengju Qian
Yu Liu

For a long time, the vision community tries to learn the spatio-temporal representation by combining convolutional neural network together with various temporal models, such as the families of Markov chain, optical ﬂow, RNN and temporal convolution. However, these pipelines consume enormous computing resources due to the alternately learning process for spatial and temporal information. One natural question is whether we can embed the temporal information into the spatial one so the information in the two domains can be jointly learned once-only. In this work, we answer this question by presenting a simple yet powerful operator – temporal interlacing network (TIN). Instead of learning the temporal features, TIN fuses the two kinds of information by interlacing spatial representations from the past to the future, and vice versa. A differentiable interlacing target can be learned to control the interlacing process. In this way, a heavy temporal model is replaced by a simple interlacing operator. We theoretically prove that with a learnable interlacing target, TIN performs equivalently to the regularized temporal convolution network (r-TCN), but gains 4% more accuracy with 6x less latency on 6 challenging benchmarks. These results push the state-of-the-art performances of video understanding by a considerable margin. Not surprising, the ensemble model of the proposed TIN won the 1st place in the ICCV19 - Multi Moments in Time challenge. Code is made available to facilitate further research. 1

PDF Details

IROS Conference 2019 Conference Paper

A Novel Small-scale Turtle-inspired Amphibious Spherical Robot

Huiming Xing
Shuxiang Guo
Liwei Shi
Xihuan Hou
Yu Liu
Huikang Liu
Yao Hu
Debin Xia

This paper describes a novel small-scale turtle-inspired Amphibious Spherical Robot (ASRobot) to accomplish exploration tasks in the restricted environment, such as amphibious areas and narrow underwater cave. A Legged, Multi-Vectored Water-Jet Composite Propulsion Mechanism (LMVWCPM) is designed with four legs, one of which contains three connecting rod parts, one water-jet thruster and three joints driven by digital servos. Using this mechanism, the robot is able to walk like amphibious turtles on various terrains and swim flexibly in submarine environment. A simplified kinematic model is established to analyze crawling gaits. With simulation of the crawling gait, the driving torques of different joints contributed to the choice of servos and the size of links of legs. Then we also modeled the robot in water and proposed several underwater locomotion. In order to assess the performance of the proposed robot, a series of experiments were carried out in the lab pool and on flat ground using the prototype robot. Experiments results verified the effectiveness of LMVWCPM and the amphibious control approaches.

Details

ICRA Conference 2019 Conference Paper

Global Localization with Object-Level Semantics and Topology

Yu Liu
Yvan R. Petillot
David Lane
Sen Wang 0002

Global localization lies at the heart of autonomous navigation and Simultaneous Localization and Mapping (SLAM). The appearance-based approach has been successful, but still faces many open challenges in environments where visual conditions vary significantly over time. In this paper, we propose an integrated solution to leverage object-level dense semantics and spatial understanding of the environment for global localization. Our approach models an environment with 3D dense semantics, semantic graph and their topology. This object-level representation is then used for place recognition via semantic object association, followed by 6-DoF pose estimation by the semantic-level point alignment. Extensive experiments show that our approach can achieve robust global localization under extreme appearance changes. It is also capable of coping with other challenging scenarios, such as dynamic environments and incomplete query observations.

Details

AAAI Conference 2019 Conference Paper

Gradient Harmonized Single-Stage Detector

Buyu Li
Yu Liu
Xiaogang Wang

Despite the great success of two-stage detectors, single-stage detector is still a more elegant and efficient way, yet suffers from the two well-known disharmonies during training, i. e. the huge difference in quantity between positive and negative examples as well as between easy and hard examples. In this work, we first point out that the essential effect of the two disharmonies can be summarized in term of the gradient. Further, we propose a novel gradient harmonizing mechanism (GHM) to be a hedging for the disharmonies. The philosophy behind GHM can be easily embedded into both classification loss function like cross-entropy (CE) and regression loss function like smooth-L1 (SL1) loss. To this end, two novel loss functions called GHM-C and GHM-R are designed to balancing the gradient flow for anchor classification and bounding box refinement, respectively. Ablation study on MS COCO demonstrates that without laborious hyper-parameter tuning, both GHM-C and GHM-R can bring substantial improvement for single-stage detector. Without any whistles and bells, the proposed model achieves 41. 6 mAP on COCO testdev set which surpass the state-of-the-art method, Focal Loss (FL) + SL1, by 0. 8. The code1 is released to facilitate future research.

PDF Details

AAAI Conference 2019 Conference Paper

Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

Hang Zhou
Yu Liu
Ziwei Liu
Ping Luo
Xiaogang Wang

Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.

PDF Details

IJCAI Conference 2018 Conference Paper

A Simple Convolutional Neural Network for Accurate P300 Detection and Character Spelling in Brain Computer Interface

Hongchang Shan
Yu Liu
Todor Stefanov

A Brain Computer Interface (BCI) character speller allows human-beings to directly spell characters using eye-gazes, thereby building communication between the human brain and a computer. Convolutional Neural Networks (CNNs) have shown better performance than traditional machine learning methods for BCI signal recognition and its application to the character speller. However, current CNN architectures limit further accuracy improvements of signal detection and character spelling and also need high complexity to achieve competitive accuracy, thereby preventing the use of CNNs in portable BCIs. To address these issues, we propose a novel and simple CNN which effectively learns feature representations from both raw temporal information and raw spatial information. The complexity of the proposed CNN is significantly reduced compared with state-of-the-art CNNs for BCI signal detection. We perform experiments on three benchmark datasets and compare our results with those in previous research works which report the best results. The comparison shows that our proposed CNN can increase the signal detection accuracy by up to 15. 61% and the character spelling accuracy by up to 19. 35%.

PDF Details

NeurIPS Conference 2018 Conference Paper

Derivative Estimation in Random Design

Yu Liu
Kris De Brabanter

We propose a nonparametric derivative estimation method for random design without having to estimate the regression function. The method is based on a variance-reducing linear combination of symmetric difference quotients. First, we discuss the special case of uniform random design and establish the estimator’s asymptotic properties. Secondly, we generalize these results for any distribution of the dependent variable and compare the proposed estimator with popular estimators for derivative estimation such as local polynomial regression and smoothing splines.

PDF Details

AAAI Conference 2018 Conference Paper

Region-Based Quality Estimation Network for Large-Scale Person Re-Identification

Guanglu Song
Biao Leng
Yu Liu
Congrui Hetang
Shaofan Cai

One of the major restrictions on the performance of videobased person re-id is partial noise caused by occlusion, blur and illumination. Since different spatial regions of a single frame have various quality, and the quality of the same region also varies across frames in a tracklet, a good way to address the problem is to effectively aggregate complementary information from all frames in a sequence, using better regions from other frames to compensate the inﬂuence of an image region with poor quality. To achieve this, we propose a novel Region-based Quality Estimation Network (RQEN), in which an ingenious training mechanism enables the effective learning to extract the complementary region-based information between different frames. Compared with other feature extraction methods, we achieved comparable results of 92. 4%, 76. 1% and 77. 83% on the PRID 2011, iLIDS-VID and MARS, respectively. In addition, to alleviate the lack of clean large-scale person re-id datasets for the community, this paper also contributes a new high-quality dataset, named “Labeled Pedestrian in the Wild (LPW)” which contains 7, 694 tracklets with over 590, 000 images. Despite its relatively large scale, the annotations also possess high cleanliness. Moreover, it’s more challenging in the following aspects: the age of characters varies from childhood to elderhood; the postures of people are diverse, including running and cycling in addition to the normal walking state.

PDF Details

AAAI Conference 2018 Conference Paper

Transferable Semi-Supervised Semantic Segmentation

Huaxin Xiao
Yunchao Wei
Yu Liu
Maojun Zhang
Jiashi Feng

The performance of deep learning based semantic segmentation models heavily depends on sufﬁcient data with careful annotations. However, even the largest public datasets only provide samples with pixel-level annotations for rather limited semantic categories. Such data scarcity critically limits scalability and applicability of semantic segmentation models in real applications. In this paper, we propose a novel transferable semi-supervised semantic segmentation model that can transfer the learned segmentation knowledge from a few strong categories with pixel-level annotations to unseen weak categories with only image-level annotations, signiﬁcantly broadening the applicable territory of deep segmentation models. In particular, the proposed model consists of two complementary and learnable components: a Label transfer Network (L-Net) and a Prediction transfer Network (P- Net). The L-Net learns to transfer the segmentation knowledge from strong categories to the images in the weak categories and produces coarse pixel-level semantic maps, by effectively exploiting the similar appearance shared across categories. Meanwhile, the P-Net tailors the transferred knowledge through a carefully designed adversarial learning strategy and produces reﬁned segmentation results with better details. Integrating the L-Net and P-Net achieves 96. 5% and 89. 4% performance of the fully-supervised baseline using 50% and 0% categories with pixel-level annotations respectively on PASCAL VOC 2012. With such a novel transfer mechanism, our proposed model is easily generalizable to a variety of new categories, only requiring image-level annotations, and offers appealing scalability in real applications.

PDF Details

AAAI Conference 2017 Conference Paper

Let Your Photos Talk: Generating Narrative Paragraph for Photo Stream via Bidirectional Attention Recurrent Neural Networks

Yu Liu
Jianlong Fu
Tao Mei
Chang Wen Chen

Automatic generation of natural language description for individual images (a. k. a. image captioning) has attracted extensive research attention. In this paper, we take one step further to investigate the generation of a paragraph to describe a photo stream for the purpose of storytelling. This task is even more challenging than individual image description due to the difﬁculty in modeling the large visual variance in an ordered photo collection and in preserving the long-term language coherence among multiple sentences. To deal with these challenges, we formulate the task as a sequence-to-sequence learning problem and propose a novel joint learning model by leveraging the semantic coherence in a photo stream. Specifically, to reduce visual variance, we learn a semantic space by jointly embedding each photo with its corresponding contextual sentence, so that the semantically related photos and their correlations are discovered. Then, to preserve language coherence in the paragraph, we learn a novel Bidirectional Attention-based Recurrent Neural Network (BARNN) model, which can attend on the discovered semantic relation to produce a sentence sequence and maintain its consistence with the photo stream. We integrate the two-step learning components into one single optimization formulation and train the network in an end-to-end manner. Experiments on three widely-used datasets (NYC/Disney/SIND) show that the proposed approach outperforms state-of-the-art methods with large margins for both retrieval and paragraph generation tasks. We also show the subjective preference of the machinegenerated stories by the proposed approach over the baselines through a user study with 40 human subjects.

PDF Details

NeurIPS Conference 2017 Conference Paper

Unsupervised Sequence Classification using Sequential Output Statistics

Yu Liu
Jianshu Chen
Li Deng

We consider learning a sequence classifier without labeled data by using sequential output statistics. The problem is highly valuable since obtaining labels in training data is often costly, while the sequential output statistics (e. g. , language models) could be obtained independently of input data and thus with low or no cost. To address the problem, we propose an unsupervised learning cost function and study its properties. We show that, compared to earlier works, it is less inclined to be stuck in trivial solutions and avoids the need for a strong generative model. Although it is harder to optimize in its functional form, a stochastic primal-dual gradient method is developed to effectively solve the problem. Experiment results on real-world datasets demonstrate that the new unsupervised learning method gives drastically lower errors than other baseline methods. Specifically, it reaches test errors about twice of those obtained by fully supervised learning.

PDF Details

NeurIPS Conference 2016 Conference Paper

Combinatorial Multi-Armed Bandit with General Reward Functions

Wei Chen
Wei Hu
Fu Li
Jian Li
Yu Liu
Pinyan Lu

In this paper, we study the stochastic combinatorial multi-armed bandit (CMAB) framework that allows a general nonlinear reward function, whose expected value may not depend only on the means of the input random variables but possibly on the entire distributions of these variables. Our framework enables a much larger class of reward functions such as the $\max()$ function and nonlinear utility functions. Existing techniques relying on accurate estimations of the means of random variables, such as the upper confidence bound (UCB) technique, do not work directly on these functions. We propose a new algorithm called stochastically dominant confidence bound (SDCB), which estimates the distributions of underlying random variables and their stochastically dominant confidence bounds. We prove that SDCB can achieve $O(\log T)$ distribution-dependent regret and $\tilde{O}(\sqrt{T})$ distribution-independent regret, where $T$ is the time horizon. We apply our results to the $K$-MAX problem and expected utility maximization problems. In particular, for $K$-MAX, we provide the first polynomial-time approximation scheme (PTAS) for its offline problem, and give the first $\tilde{O}(\sqrt T)$ bound on the $(1-\epsilon)$-approximation regret of its online problem, for any $\epsilon>0$.

PDF Details

ICML Conference 2016 Conference Paper

K-Means Clustering with Distributed Dimensions

Hu Ding 0003
Yu Liu
Lingxiao Huang
Jian Li 0015

Distributed clustering has attracted significant attention in recent years. In this paper, we study the k-means problem in the distributed dimension setting, where the dimensions of the data are partitioned across multiple machines. We provide new approximation algorithms, which incur low communication costs and achieve constant approximation ratios. The communication complexity of our algorithms significantly improve on existing algorithms. We also provide the first communication lower bound, which nearly matches our upper bound in a certain range of parameter setting. Our experimental results show that our algorithms outperform existing algorithms on real data-sets in the distributed dimension setting.

Details

IROS Conference 2006 Conference Paper

Person-Tracking with Occlusion Using Appearance Filters

Peng Bai
Hong Qiao
Anhua Wan
Yu Liu

To deal with the problem of tracking person with a mobile robot in dynamic scene with occlusion, a tracking system is presented in this paper. We focus on the individual person tracking system, which is applied on our two-wheel mobile robot with a single PTZ (pan-tilt-zoom) camera. The tracking system works reliably in the dynamic environment with partial/complete occlusion. For the sake of complete occlusion, we use several filters and spatial relation restriction to represent the target model. These filters are all based on the appearance of the target, and each filter models a part of human body which is more rigid than the entire body. To construct a tracking system, the situation-based strategy and a simple frame-to-frame tracker are involved in the system, and an improved mean-shift tracking algorithm is applied as the tracker in this paper. Finally, experimental results on tracking individual person with complete occlusion in different environments are shown, which demonstrate the robustness and effectiveness of the algorithm

Details

IROS Conference 2006 Conference Paper

Resource Management with RFID Technology in Automatic Warehouse System

Guidong Liu
Wensheng Yu
Yu Liu

Warehouse operation has become a critical activity in supply chain to outperform competitors on customer service, lead times and costs. However, managers often meet difficulties in formulating good resource utilization plan due to shortage of comprehensive resource information. This paper proposes a resource management system with RFID technology, which can access to the real-time position and operation process information of material handling equipments, as well as some load item information. This warehouse management system is designed to select the most suitable resource usage package for handling warehouse operation orders. An experiment warehouse is built in the Institute of Automation, Chinese Academy of Sciences to validate the effectiveness of the warehouse management system. The results show increase in rack space utilization and loading speed as well as decrease in work-related errors and operation costs in comparison with a similar system without RFID technology

Details