Author name cluster

Jingyu Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

27 papers

1 author row

AAAI Conference 2026 Short Paper

AEFGL: Reverse Auction and Value Evaluation-Based Federated Graph Learning Incentive Mechanism (Student Abstract)

Xin Chang
Lixin Liu
Jingyu Wang
Jinling Yu
Xiaolin Zhang

Federated Graph Learning enables multiple clients to collaboratively train graph models while protecting local private data. However, most studies have assumed that all clients contribute data voluntarily and actively. Without reasonable incentives, clients are often reluctant to contribute personal data for model training. Furthermore, the budget for incentives is limited, and if clients with low-quality graph data are incentivized to participate in training, it will negatively impact the training performance of all parties in the system. To address this, we propose AEFGL, a Reverse Auction and Value Evaluation-Based Incentive Mechanism for Federated Graph Learning. First, we design a reverse auction mechanism combining graph structural attribute motifs with client production value. Then, we propose a method for evaluating client production value based on the comparison of the client's expected reward and actual value. This mechanism can incentivize clients with high-quality graph data to participate in training within budget constraints, thereby improving the model quality. Experimental results validate the superiority of the AEFGL mechanism and the economic properties it satisfies.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Bridging the Tokenizer Gap: Semantics and Distribution-aware Knowledge Transfer for Unbiased Cross-Tokenizer Distillation

Huazheng Wang
Yongcheng Jing
Haifeng Sun
Jingyu Wang
Jianxin Liao
Leszek Rutkowski
Dacheng Tao

Cross-tokenizer knowledge distillation, where the teacher and student employ different tokenizers, is becoming increasingly prevalent, yet it poses underexplored challenges: existing methods fail to capture the rich knowledge encoded in teacher logits, as evidenced by the neglect of semantic information, inaccurate and biased logit alignment, and discarding distributional structure—ultimately leading to unfavorable distillation. To address these issues, we propose SeDi, a semantics and distribution-aware knowledge transfer framework tailored for cross-tokenizer distillation. To preserve factual knowledge, SeDi employs bipartite graph-based alignment at the tokenization level and a sliding window re-encoding strategy at the vocabulary level, enabling unbiased transfer of the teacher’s next-token predictions into the student’s vocabulary space. To further retain distributional information, we align the student’s entropy with that of the teacher by incorporating the student’s own logits during training, which helps to mitigate the exposure bias problem. Experiments on ten datasets across three task domains and five different teacher-student model pairs with varying vocabulary sizes demonstrate that SeDi delivers substantial improvements, with gains of up to 19.8%.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

A³-Net: Calibration-Free Multi-View 3D Hand Reconstruction for Enhanced Musical Instrument Learning

Geng Chen
Xufeng Jian
Yuchen Chen
Pengfei Ren
Jingyu Wang
Haifeng Sun
Qi Qi
Jing Wang

Precise 3D hand posture is essential for learning musical instruments. Reconstructing highly precise 3D hand gestures enables learners to correct and master proper techniques through 3D simulation and Extended Reality. However, exsiting methods typically rely on precisely calibrated multi-camera systems, which are not easily deployable in everyday environments. In this paper, we focus on calibration-free multi-view 3D hand reconstruction in unconstrained scenarios. Establishing correspondences between multi-view images is particularly challenging without camera extrinsics. To address this, we propose A^3-Net, a multi-level alignment framework that utilizes 3D structural representations with hierarchical geometric and explicit semantic information as alignment proxies, facilitating multi-view feature interaction in both 3D geometric space and 2D visual space. Specifically, we first perfrom global geometric alignment to map multi-view features into a canonical space. Subsequently, we aggregate information into predefined sparse and dense proxies to further integrate cross-view semantics through mutual interaction. Finnaly, we perfrom 2D alignment to align projected 2D visual features with 2D observations. Our method achieves state-of-the-art results in the multi-view 3D hand reconstruction task, demonstrating the effectiveness of our proposed framework.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

Beyond Statistical Analysis: Multimodal Framework for Time Series Forecasting with LLM-Driven Temporal Pattern

Jiahong Xiong
Chengsen Wang
Haifeng Sun
Yuhan Jing
Qi Qi
Zirui Zhuang
Lei Zhang
Jianxin Liao

Accurate forecasting of time series is crucial for many applications in the real world. Conventional methods primarily rely on statistical analysis of historical data, often leading to overfitting and failing to account for background information and constraints imposed by external events. Therefore, introducing large language models (LLMs) with robust textual capabilities holds significant potential. However, due to the inherent limitations of LLMs in handling numerical data, they do not exhibit advantages in precise numerical prediction tasks. Therefore, we propose a framework to integrate LLMs with conventional methods synergistically. Rather than directly outputting numerical predictions, we leverage the capabilities of the LLMs to generate textual temporal patterns, thereby fully utilizing their inherent knowledge and reasoning abilities. Additionally, we introduce a memory network designed to decode these textual representations into a format that numerical models can effectively interpret. This approach not only capitalizes on the strengths of the LLM in text processing but also bridges the gap between textual and numerical data, enhancing the overall predictive performance of the model. Our experimental results demonstrate the framework's effectiveness, achieving state-of-the-art performance on various benchmark datasets.

PDF Details DOI

AAAI Conference 2025 Conference Paper

ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data

Chengsen Wang
Qi Qi
Jingyu Wang
Haifeng Sun
Zirui Zhuang
Jinming Wu
Lei Zhang
Jianxin Liao

Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns

Menghao Zhang
Huazheng Wang
Pengfei Ren
Kangheng Lin
Qi Qi
Haifeng Sun
Zirui Zhuang
Lei Zhang

Large Vision-Language Models (LVLMs) pretrained on large-scale multimodal data have shown promising capabilities in Video Anomaly Detection (VAD). However, their ability to reason about abnormal events based on scene semantics remains underexplored. In this paper, we investigate LVLMs’ behavior in VAD from a visual-textual co-occurrence perspective, focusing on whether their decisions are driven by statistical shortcuts between visual instances and textual phrases. By analyzing visual-textual co-occurrence in pretraining data and conducting experiments under different data settings, we reveal a hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality or abnormality, leading to incorrect predictions when these high-frequency objects appear in semantically mismatched contexts. To address this issue, we propose VAD-DPO, a direct preference optimization method supervised with counter-example pairs. By constructing visually similar but semantically contrasting video clips, VAD-DPO encourages the model to align its predictions with the semantics of scene rather than relying on co-occurrence patterns. Extensive experiments on six benchmark datasets demonstrate the effectiveness of VAD-DPO in enhancing both anomaly detection and reasoning performance, particularly in scene-dependent scenarios.

PDF Details

EAAI Journal 2025 Journal Article

Efficient generation of power system topology diagrams based on Graph Neural Network

Chen Yang
Shengyang Wu
Tao Liu
Yixuan He
Jingyu Wang
Dongyuan Shi

Power system topology diagrams illustrate the physical and spatial relationship of system nodes and are widely used as a basic tool for displaying system structure. Well-presented topology diagrams provide better situational awareness for the operators, but their efficient generation remains a challenge. Existing approaches struggle to find a balance between visual aesthetics and the generation speed of the diagram. With the rapid changes in power system topology, there is a higher demand for the rendering speed of the graph data. To satisfy both the real-time requirement and the aesthetic quality, this paper proposes an integrated framework for efficiently generating power system topology diagrams. It consists of a Graph Neural Network (GNN) model and a graph fine-tuning model. This framework can directly optimize the raw topology diagram while preserving the relative positions of nodes in the initial layout. It achieves a decent trade-off between layout quality and computational expenses, enabling the generation of aesthetically satisfactory diagrams in a short time. Due to the strong generalization ability of GNN, the proposed model can be trained on small system datasets and used for inference on large systems. Case studies verify that the proposed GNN model can optimize the aesthetic metrics of topology diagram layouts within seconds to an average value of 0. 55. Finally, it can be used in power system applications as a fundamental tool for topology diagram generation and optimization.

Details DOI

IJCAI Conference 2025 Conference Paper

Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU

Shuxi Guo
Zikang Xu
Jiahao Liu
Jinyi Zhang
Qi Qi
Haifeng Sun
Jun Huang
Jianxin Liao

Deep learning-based recommendation systems are increasingly important in the industry. To meet strict SLA requirements, serving frameworks must efficiently handle concurrent queries. However, current serving systems fail to serve concurrent queries due to the following problems: (1) inefficient operator (op) scheduling due to the query-wise op launching mechanism, and (2) heavy contention caused by the mutable nature of recommendation model inference. This paper presents RecOS, a system designed to optimize concurrent recommendation model inference on GPUs. RecOS efficiently schedules ops from different queries by monitoring GPU workloads and assigning ops to the most suitable streams. This approach reduces contention and enhances inference efficiency by leveraging inter-op parallelism and op characteristics. To maintain correctness across multiple CUDA streams, RecOS introduces a unified asynchronous tensor management mechanism. Evaluations demonstrate that RecOS improves online service performance, reducing latency by up to 68%.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Generalizable Hand-Object Modeling from Monocular RGB Images via 3D Gaussians

Xingyu Liu
Pengfei Ren
Qi Qi
Haifeng Sun
Zirui Zhuang
Jing Wang
Jianxin Liao
Jingyu Wang

Recent advances in hand-object interaction modeling have employed implicit representations, such as Signed Distance Functions (SDF) and Neural Radiance Fields (NeRF) to reconstruct hands and objects with arbitrary topology and photo-realistic detail. However, these methods often rely on dense 3D surface annotations, or are tailored to short clips constrained in motion trajectories and scene contexts, limiting their generalization to diverse environments and movement patterns. In this work, we present HOGS, an adaptively perceptive 3D Gaussian Splatting (3DGS) framework for generalizable hand-object modeling from unconstrained monocular RGB images. By integrating photometric cues from the visual modality with the physically grounded structure of 3D Gaussians, HOGS disentangles inherent geometry from transient lighting and motion-induced appearance changes. This endows hand-object assets with the ability to generalize to unseen environments and dynamic motion patterns. Experiments on two challenging datasets demonstrate that HOGS outperforms state-of-the-art methods in monocular hand-object reconstruction and photo-realistic rendering.

PDF Details

NeurIPS Conference 2025 Conference Paper

Perception-R1: Pioneering Perception Policy with Reinforcement Learning

En Yu
Kangheng Lin
Liang Zhao
jisheng yin
Yana Wei
Yuang Peng
Haoran Wei
Jianjian Sun

Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual perplexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approaching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2-VL-2B-Instruct, Perception-R1 achieves +4. 2% on RefCOCO+, +17. 9% on PixMo-Count, +4. 2% on PageOCR, and notably, 31. 9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.

PDF Details

AAAI Conference 2025 Conference Paper

Seeing Beyond Noise: Joint Graph Structure Evaluation and Denoising for Multimodal Recommendation

Yuxin Qi
Quan Zhang
Xi Lin
Xiu Su
Jiani Zhu
Jingyu Wang
Jianhua Li

Multimodal Recommendation Systems (MRSs) boost traditional user-item interaction-based methods by incorporating multimodal information. However, existing methods ignore the inherent noise brought by (1) noisy semantic priors in multimodal content, and (2) noisy user interactions in history records, therefore diminishing model performance. To fill this gap, we propose to denoise MRSs by jointly EValuating structure Effectiveness and mitigating Noisy links (EVEN). Firstly, for semantic prior noise in multimodal content, EVEN builds item homogeneous consistency and denoises it by evaluating behavior-driven confidence. Secondly, for noise in user interactions, EVEN updates user feedback by denoising observed interactions following implicit contribution evaluation of high-order representations. Thirdly, EVEN performs cross-modal alignment through self-guided structure learning, reinforcing task-specific inter-modal dependency modeling and cross-modal fusion. Through extensive experiments on three widely-used datasets, EVEN achieves an average improvement of 8.95% and 5.90% in recommendation accuracy compared with LGMRec and FREEDOM, respectively, without extending the total training time.

PDF Details DOI

EAAI Journal 2025 Journal Article

Three-dimensional reconstruction image generation of traditional Chinese painting elements

Qiyao Hu
Jingyu Wang
Xianlin Peng
Tengfei Li
Rui Cao

This paper presents a comprehensive pipeline for generating detailed three-dimensional (3D) models from single images of traditional Chinese painting elements. This task is particularly challenging due to the lack of 3D datasets for Chinese paintings and the limited research on their 3D reconstruction. As a result, direct access to multiple views is precluded. We propose a novel method for the 3D reconstruction of Traditional Chinese Painting Elements, termed TCPE-3D, which has three components of the process. The first component is a multi-view synthesis module named One To Six (OTX) - Multi-View Generating (MVG) Module. This module creates six fixed-view images through a series of preprocessing steps. These images are used to generate the Local Light Field Fusion (LLFF) dataset within the Neural Radiance Fields (NeRF) synthesis module. This process leads to the creation of detailed mesh structures in the final Mesh Generation module. Comparison with several state-of-the-art 3D reconstruction methods shows that our framework achieves better visualization results and higher technical specifications. Additionally, it solves the Janus problem encountered by other algorithms for Chinese painting data. Our dataset is available at https: //github. com/LPDLG/3DTCP-Dataset.

Details DOI

NeurIPS Conference 2025 Conference Paper

Unified 2D-3D Discrete Priors for Noise-Robust and Calibration-Free Multiview 3D Human Pose Estimation

Geng Chen
Pengfei Ren
Xufeng Jian
Haifeng Sun
Menghao Zhang
Qi Qi
Zirui Zhuang
Jing Wang

Multi-view 3D human pose estimation (HPE) leverages complementary information across views to improve accuracy and robustness. Traditional methods rely on camera calibration to establish geometric correspondences, which is sensitive to calibration accuracy and lacks flexibility in dynamic settings. Calibration-free approaches address these limitations by learning adaptive view interactions, typically leveraging expressive and flexible continuous representations. However, as the multiview interaction relationship is learned entirely from data without constraint, they are vulnerable to noisy input, which can propagate, amplify and accumulate errors across all views, severely corrupting the final estimated pose. To mitigate this, we propose a novel framework that integrates a noise-resilient discrete prior into the continuous representation-based model. Specifically, we introduce the \textit{UniCodebook}, a unified, compact, robust, and discrete representation complementary to continuous features, allowing the model to benefit from robustness to noise while preserving regression capability. Furthermore, we further propose an attribute-preserving and complementarity-enhancing Discrete-Continuous Spatial Attention (DCSA) mechanism to facilitate interaction between discrete priors and continuous pose features. Extensive experiments on three representative datasets demonstrate that our approach outperforms both calibration-required and calibration-free methods, achieving state-of-the-art performance.

PDF Details

EAAI Journal 2024 Journal Article

A comprehensive end-to-end computer vision framework for restoration and recognition of low-quality engineering drawings

Lvyang Yang
Jiankang Zhang
Huaiqiang Li
Longfei Ren
Chen Yang
Jingyu Wang
Dongyuan Shi

The digitization of engineering drawings is crucial for efficient reuse, distribution, and archiving. Existing computer vision approaches for digitizing engineering drawings typically assume the input drawings have high quality. However, in reality, engineering drawings are often blurred and distorted due to improper scanning, storage, and transmission, which may jeopardize the effectiveness of existing approaches. This paper focuses on restoring and recognizing low-quality engineering drawings, where an end-to-end framework is proposed to improve the quality of the drawings and identify the graphical symbols on them. The framework uses K-means clustering to classify different engineering drawing patches into simple and complex texture patches based on their gray level co-occurrence matrix statistics. Computer vision operations and a modified Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) model are then used to improve the quality of the two types of patches, respectively. A modified Faster Region-based Convolutional Neural Network (Faster R-CNN) model is used to recognize the quality-enhanced graphical symbols. Additionally, a multi-stage task-driven collaborative learning strategy is proposed to train the modified ESRGAN and Faster R-CNN models to improve the resolution of engineering drawings in the direction that facilitates graphical symbol recognition, rather than human visual perception. A synthetic data generation method is also proposed to construct quality-degraded samples for training the framework. Experiments on real-world electrical diagrams show that the proposed framework achieves an accuracy of 98. 98% and a recall of 99. 33%, demonstrating its superiority over previous approaches. Moreover, the framework is integrated into a widely-used power system software application to showcase its practicality. The reference codes and data can be found at https: //github. com/Lattle-y/AI-recognition-for-lq-ed. git Future work will focus on improving the generalizability of the proposed framework to different quality degradation scenarios and extrapolating the application to different engineering domains.

Details DOI

NeurIPS Conference 2024 Conference Paper

FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models

Wanyi Ning
Jingyu Wang
Qi Qi
Mengde Zhu
Haifeng Sun
Daixuan Cheng
Jianxin Liao
Ce Zhang

Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users to cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre-trained models. To this end, we propose a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models. Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20B family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation

Xingyu Liu
Pengfei Ren
Yuanyuan Gao
Jingyu Wang
Haifeng Sun
Qi Qi
Zirui Zhuang
Jianxin Liao

Previous 3D hand pose estimation methods primarily rely on a single modality, either RGB or depth, and the comprehensive utilization of the dual modalities has not been extensively explored. RGB and depth data provide complementary information and thus can be fused to enhance the robustness of 3D hand pose estimation. However, there exist two problems for applying existing fusion methods in 3D hand pose estimation: redundancy of dense feature fusion and ambiguity of visual features. First, pixel-wise feature interactions introduce high computational costs and ineffective calculations of invalid pixels. Second, visual features suffer from ambiguity due to color and texture similarities, as well as depth holes and noise caused by frequent hand movements, which interferes with modeling cross-modal correlations. In this paper, we propose Keypoint-Fusion for RGB-D based 3D hand pose estimation, which leverages the unique advantages of dual modalities to mutually eliminate the feature ambiguity, and performs cross-modal feature fusion in a more efficient way. Specifically, we focus cross-modal fusion on sparse yet informative spatial regions (i.e. keypoints). Meanwhile, by explicitly extracting relatively more reliable information as disambiguation evidence, depth modality provides 3D geometric information for RGB feature pixels, and RGB modality complements the precise edge information lost due to the depth noise. Keypoint-Fusion achieves state-of-the-art performance on two challenging hand datasets, significantly decreasing the error compared with previous single-modal methods.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Rethinking the Power of Timestamps for Robust Time Series Forecasting: A Global-Local Fusion Perspective

Chengsen Wang
Qi Qi
Jingyu Wang
Haifeng Sun
Zirui Zhuang
Jinming Wu
Jianxin Liao

Time series forecasting has played a pivotal role across various industries, including finance, transportation, energy, healthcare, and climate. Due to the abundant seasonal information they contain, timestamps possess the potential to offer robust global guidance for forecasting techniques. However, existing works primarily focus on local observations, with timestamps being treated merely as an optional supplement that remains underutilized. When data gathered from the real world is polluted, the absence of global information will damage the robust prediction capability of these algorithms. To address these problems, we propose a novel framework named GLAFF. Within this framework, the timestamps are modeled individually to capture the global dependencies. Working as a plugin, GLAFF adaptively adjusts the combined weights for global and local information, enabling seamless collaboration with any time series forecasting backbone. Extensive experiments conducted on nine real-world datasets demonstrate that GLAFF significantly enhances the average performance of widely used mainstream forecasting models by 12. 5\%, surpassing the previous state-of-the-art method by 5. 5\%.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Safeguarding Sustainable Cities: Unsupervised Video Anomaly Detection through Diffusion-based Latent Pattern Learning

Menghao Zhang
Jingyu Wang
Qi Qi
Pengfei Ren
Haifeng Sun
Zirui Zhuang
Lei Zhang
Jianxin Liao

Sustainable cities requires high-quality community management and surveillance analytics, which are supported by video anomaly detection techniques. However, mainstream video anomaly detection techniques still require manually labeled data and do not apply to real-world massive videos. Without labeling, unsupervised video anomaly detection (UVAD) is challenged by the problem of pseudo-labeled noise and the openness of anomaly detection. In response, a diffusion-based latent pattern learning UVAD framework is proposed, called DiffVAD. The method learns potential patterns by generating different patterns of the same event through diffusion models. The detection of anomalies is realized by evaluating the pattern distribution. The different patterns of normal events are diverse but correlated, while the different patterns of abnormal events are more diffuse. This manner of detection is equally effective for unseen normal events in the training set. In addition, we design a refinement strategy for pseudo-labels to mitigate the effects of the noise problem. Extensive experiments on six benchmark datasets demonstrate the design’s promising generalization ability and high efficiency. Specifically, DiffVAD obtains an AUC score of 81. 9% on the ShanghaiTech dataset.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection

Chengsen Wang
Zirui Zhuang
Qi Qi
Jingyu Wang
Xingyu Wang
Haifeng Sun
Jianxin Liao

Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead to numerous false alarms. We propose **D**ynamic **D**ecomposition with **D**iffusion **R**econstruction (D$^3$R), a novel anomaly detection network for real-world unstable data to fill the gap. D$^3$R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix-attention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, we control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments on various real-world datasets demonstrate that D$^3$R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models.

PDF Details

EAAI Journal 2023 Journal Article

Feature pre-inpainting enhanced transformer for video inpainting

Guanxiao Li
Ke Zhang
Yu Su
Jingyu Wang

Transformer-based video inpainting methods aggregate coherent contents into missing regions by learning dependencies spatial–temporally. However, existing methods suffer from the inaccurate self-attention calculation and excessive quadratic computational complexity, due to uninformative representations of missing regions and inefficient global self-attention mechanisms, respectively. To mitigate these problems, we propose a Feature pre-Inpainting enhanced Transformer (FITer) video inpainting method, in which the feature pre-inpainting network (FPNet) and local–global interleaving Transformer are designed. The FPNet pre-inpaints missing features before the Transformer by exploiting spatial context, and the representations of missing regions are thus enhanced with more informative content. Therefore, the interleaving Transformer can calculate more accurate self-attention weights and learns more effective dependencies between missing and valid regions. Since the interleaving Transformer involves both global and window-based local self-attention mechanisms, the proposed FITer method can effectively aggregate spatial–temporal features into missing regions while improving efficiency. Experiments on YouTube-VOS and DAVIS datasets demonstrate that the FITer method outperforms previous methods qualitatively and quantitatively.

Details DOI

IJCAI Conference 2023 Conference Paper

Not Only Pairwise Relationships: Fine-Grained Relational Modeling for Multivariate Time Series Forecasting

Jinming Wu
Qi Qi
Jingyu Wang
Haifeng Sun
Zhikang Wu
Zirui Zhuang
Jianxin Liao

Recent graph-based methods achieve significant success in multivariate time series modeling and forecasting due to their ability to handle relationships among time series variables. However, only pairwise relationships are considered in most existing works. They ignore beyond-pairwise relationships and their potential categories in practical scenarios, which leads to incomprehensive relationship learning for multivariate time series forecasting. In this paper, we present ReMo, a Relational Modeling-based method, to promote fine-grained relational learning among multivariate time series data. Firstly, by treating time series variables and complex relationships as nodes and hyperedges, we extract multi-view hypergraphs from data to capture beyond-pairwise relationships. Secondly, a novel hypergraph message passing strategy is designed to characterize both nodes and hyperedges by inferring the potential categories of relationships and further distinguishing their impacts on time series variables. By integrating these two modules into the time series forecasting framework, ReMo effectively improves the performance of multivariate time series forecasting. The experimental results on seven commonly used datasets from different domains demonstrate the superiority of our model.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision

Ce Ge
Jingyu Wang
Qi Qi
Haifeng Sun
Tong Xu
Jianxin Liao

The sketch-based image retrieval (SBIR) task has long been researched at the instance level, where both query sketches and candidate images are assumed to contain only one dominant object. This strong assumption constrains its application, especially with the increasingly popular intelligent terminals and human-computer interaction technology. In this work, a more general scene-level SBIR task is explored, where sketches and images can both contain multiple object instances. The new general task is extremely challenging due to several factors: (i) scene-level SBIR inherently shares sketch-specific difficulties with instance-level SBIR (e.g., sparsity, abstractness, and diversity), (ii) the cross-modal similarity is measured between two partially aligned domains (i.e., not all objects in images are drawn in scene sketches), and (iii) besides instance-level visual similarity, a more complex multi-dimensional scene-level feature matching problem is imposed (including appearance, semantics, layout, etc.). Addressing these challenges, a novel Conditional Graph Autoencoder model is proposed to deal with scene-level sketch-images retrieval. More importantly, the model can be trained with only pairwise supervision, which distinguishes our study from others in that elaborate instance-level annotations (for example, bounding boxes) are no longer required. Extensive experiments confirm the ability of our model to robustly retrieve multiple related objects at the scene level and exhibit superior performance beyond strong competitors.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval

Ce Ge
Jingyu Wang
Qi Qi
Haifeng Sun
Tong Xu
Jianxin Liao

Sketch-based image retrieval (SBIR) is an attractive research area where freehand sketches are used as queries to retrieve relevant images. Existing solutions have advanced the task to the challenging zero-shot setting (ZS-SBIR), where the trained models are tested on new classes without seen data. However, they are prone to overfitting under a realistic scenario when the test data includes both seen and unseen classes. In this paper, we study generalized ZS-SBIR (GZS-SBIR) and propose a novel semi-transductive learning paradigm. Transductive learning is performed on the image modality to explore the potential data distribution within unseen classes, and zero-shot learning is performed on the sketch modality sharing the learned knowledge through a semi-heterogeneous architecture. A hybrid metric learning strategy is proposed to establish semantics-aware ranking property and calibrate the joint embedding space. Extensive experiments are conducted on two large-scale benchmarks and four evaluation metrics. The results show that our method is superior over the state-of-the-art competitors in the challenging GZS-SBIR task.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Two Heads Are Better than One: Image-Point Cloud Network for Depth-Based 3D Hand Pose Estimation

Pengfei Ren
Yuchen Chen
Jiachang Hao
Haifeng Sun
Qi Qi
Jingyu Wang
Jianxin Liao

Depth images and point clouds are the two most commonly used data representations for depth-based 3D hand pose estimation. Benefiting from the structuring of image data and the inherent inductive biases of the 2D Convolutional Neural Network (CNN), image-based methods are highly efficient and effective. However, treating the depth data as a 2D image inevitably ignores the 3D nature of depth data. Point cloud-based methods can better mine the 3D geometric structure of depth data. However, these methods suffer from the disorder and non-structure of point cloud data, which is computationally inefficient. In this paper, we propose an Image-Point cloud Network (IPNet) for accurate and robust 3D hand pose estimation. IPNet utilizes 2D CNN to extract visual representations in 2D image space and performs iterative correction in 3D point cloud space to exploit the 3D geometry information of depth data. In particular, we propose a sparse anchor-based "aggregation-interaction-propagation'' paradigm to enhance point cloud features and refine the hand pose, which reduces irregular data access. Furthermore, we introduce a 3D hand model to the iterative correction process, which significantly improves the robustness of IPNet to occlusion and depth holes. Experiments show that IPNet outperforms state-of-the-art methods on three challenging hand datasets.

PDF Details DOI

EAAI Journal 2022 Journal Article

MRRNet: Learning multiple region representation for video person re-identification

Hui Fu
Ke Zhang
Haoyu Li
Jingyu Wang

Video person re-identification is a crucial component of a robust surveillance system. Within a video clip, different human regions exhibit unique stability characteristics, which would be harmful to generating a discriminative representation. Unfortunately, prior works cannot effectively deal with the stability characteristics of different regions. To tackle this problem, we propose a Multiple Region Representation Network (MRRNet) that aims to discover the discriminative information from different human regions. Firstly, a Stable Region Representation (SRR) layer is proposed to capture important clues from the stable regions and exchange temporal information by cross-relation aware operation. Secondly, a Multiple Region Representation (MRR) layer is designed to address the unstable regions and preserve the attention on stable regions. Thirdly, SRR and MRR can be conveniently inserted into multiple stages of the deep residual networks and significantly improve the performance of the network. Comprehensive experiments validate the effectiveness of our network. Particularly, MRRNet achieves 86. 7% mAP and 91. 1% Rank-1 accuracy on the MARS dataset, which outperforms state-of-the-arts.

Details DOI

AAAI Conference 2021 Conference Paper

Question-Driven Span Labeling Model for Aspect–Opinion Pair Extraction

Lei Gao
Yulong Wang
Tongcun Liu
Jingyu Wang
Lei Zhang
Jianxin Liao

Aspect term extraction and opinion word extraction are two fundamental subtasks of aspect-based sentiment analysis. The internal relationship between aspect terms and opinion words is typically ignored, and information for the decisionmaking of buyers and sellers is insufficient. In this paper, we explore an aspect–opinion pair extraction (AOPE) task and propose a Question-Driven Span Labeling (QDSL) model to extract all the aspect–opinion pairs from user-generated reviews. Specifically, we divide the AOPE task into aspect term extraction (ATE) and aspect-specified opinion extraction (ASOE) subtasks; we first extract all the candidate aspect terms and then the corresponding opinion words given the aspect term. Unlike existing approaches that use the BIObased tagging scheme for extraction, the QDSL model adopts a span-based tagging scheme and builds a question–answerbased machine-reading comprehension task for an effective aspect–opinion pair extraction. Extensive experiments conducted on three tasks (ATE, ASOE, and AOPE) on four benchmark datasets demonstrate that the proposed method significantly outperforms state-of-the-art approaches.

PDF Details

AAAI Conference 2020 Conference Paper

AWR: Adaptive Weighting Regression for 3D Hand Pose Estimation

Weiting Huang
Pengfei Ren
Jingyu Wang
Qi Qi
Haifeng Sun

In this paper, we propose an adaptive weighting regression (AWR) method to leverage the advantages of both detectionbased and regression-based method. Hand joint coordinates are estimated as discrete integration of all pixels in dense representation, guided by adaptive weight maps. This learnable aggregation process introduces both dense and joint supervision that allows end-to-end training and brings adaptability to weight maps, making network more accurate and robust. Comprehensive exploration experiments are conducted to validate the effectiveness and generality of AWR under various experimental settings, especially its usefulness for different types of dense representation and input modality. Our method outperforms other state-of-the-art methods on four publicly available datasets, including NYU, ICVL, MSRA and HANDS 2017 dataset.

PDF Details