Yu Ding

YNICL Journal 2025 Journal Article

Early cortical alterations and neuropsychological mechanisms in amyotrophic lateral sclerosis

• Qianqian Zhang
• Yu Ding
• Yu Zhang
• Qingyang Li
• Shiyu Shi
• Yaxi Liu
• Sijie Chen
• Qian Wu

OBJECTIVE: This study investigates the characteristics of cortical structural and functional alterations in amyotrophic lateral sclerosis (ALS) patients and their modulation of emotional and cognitive functions, as well as to discuss their diagnostic value in early-stage ALS. METHODS: Fifty-nine ALS patients (28 in ALS 1 and 31 in ALS 2, categorized using King's College Staging) and 31 healthy controls were evaluated using multiparametric MRI, motor and neuropsychological assessments, and serum neurofilament light chain (NfL) levels. Mediation analyses were performed to examine how cortical alterations influence the relationship between emotional and cognitive functions. Support vector machine (SVM) classification models were constructed to assess the diagnostic utility of differential cortical parameters. RESULTS: ALS 1 patients exhibited increased cortical thickness (CT) and functional activity in the cingulate and frontotemporal regions, correlating with neuropsychological performance and NfL levels. Mediation analysis revealed that perigenual and frontotemporal functional activity significantly modulated the relationship between depressive symptoms and cognitive function. SVM classification showed that the combined altered regions with Amplitude of Low Frequency Fluctuations (ALFF) model achieved slightly better performance (AUC = 0.853, 95 %CI: 0.687-1.000, p < 0.001) compared to CT (AUC = 0.779, 95 %CI: 0.587-0.972, p < 0.001), although both models showed limited efficacy in differentiating between ALS 1 and ALS 2 groups. CONCLUSIONS: Cortical structural and functional alterations in ALS mediate the impact of depression on cognitive function, offering insights into the neuropsychological mechanisms of the disease and potential biomarkers for early-stage diagnosis.

EAAI Journal 2024 Journal Article

A rail defect detection framework under class-imbalanced conditions based on improved you only look once network

• Yu Ding
• Qin Zhao
• Tianhao Li
• Chen Lu
• Laifa Tao
• Jian Ma

In real rail operations, defects that can lead to serious accidents occur at very low frequencies, resulting in sample scarcity and class imbalances in rail defect datasets. Under imbalanced conditions, rail defect detection models tend to be biased toward majority classes and ignore minority classes, which further leads to inaccurate defect detection results. Therefore, a two-stage rail defect detection framework based on a latent diffusion model (LDM) and an improved You Only Look Once (YOLO) network operating under imbalanced conditions is proposed. This framework aims to enhance the detection performance achieved on imbalanced defect datasets through data augmentation and model improvements. First, the LDM is used to generate many defects with extremely small sample sizes and provide high-quality generated samples to expand the original imbalanced dataset. Furthermore, a coordinate attention module and a feature fusion module are integrated into the original YOLO version 8 (YOLOv8) model to improve its detection capabilities on imbalanced datasets. The coordinate attention mechanism enhances its focus on the positional information of various defects, whereas the feature fusion module enhances its ability to fuse the multiscale features of different defects. The results of the case study demonstrate that sample generation and filtration can provide high-quality samples for dataset augmentation purposes, alleviating the impact of minority defects on the overall detection accuracy. The results of the comparison and ablation experiments show that the improved YOLOv8 model has better detection performance than that of the comparison methods due to its introduction of coordinate attention and feature fusion modules.

AAAI Conference 2024 Conference Paper

Say Anything with Any Style

• Shuai Tan
• Bin Ji
• Yu Ding
• Ye Pan

Generating stylized talking head with diverse head motions is crucial for achieving natural-looking videos but still remains challenging. Previous works either adopt a regressive method to capture the speaking style, resulting in a coarse style that is averaged across all training data, or employ a universal network to synthesize videos with different styles which causes suboptimal performance. To address these, we propose a novel dynamic-weight method, namely Say Anything with Any Style (SAAS), which queries the discrete style representation via a generative model with a learned style codebook. Specifically, we develop a multi-task VQ-VAE that incorporates three closely related tasks to learn a style codebook as a prior for style extraction. This discrete prior, along with the generative model, enhances the precision and robustness when extracting the speaking styles of the given style clips. By utilizing the extracted style, a residual architecture comprising a canonical branch and style-specific branch is employed to predict the mouth shapes conditioned on any driving audio while transferring the speaking style from the source to any desired one. To adapt to different speaking styles, we steer clear of employing a universal network by exploring an elaborate HyperStyle to produce the style-specific weights offset for the style branch. Furthermore, we construct a pose generator and a pose codebook to store the quantized pose representation, allowing us to sample diverse head motions aligned with the audio and the extracted style. Experiments demonstrate that our approach surpasses state-of-the-art methods in terms of both lip-synchronization and stylized expression. Besides, we extend our SAAS to video-driven style editing field and achieve satisfactory performance as well.

EAAI Journal 2023 Journal Article

Conditional probability based multi-objective cooperative task assignment for heterogeneous UAVs

• Xiaohua Gao
• Lei Wang
• Xinyong Yu
• Xichao Su
• Yu Ding
• Chen Lu
• Haijun Peng
• Xinwei Wang

In actual air combat, there is an inevitable risk that an unmanned aerial vehicle (UAV) will be destroyed. However, this risk is rarely considered in the mission planning phase. In this paper, we focus on cooperative mission assignment for heterogeneous UAVs. We develop a multi-objective optimization model to find a balance between mission gains and UAV losses. The objective function is expressed using conditional probability theory by introducing the probabilities of mission success and UAV loss. Munitions loading capacity, time constraints, and priority constraints are modeled as constraints. To solve this combinatorial problem, an improved multi-objective genetic algorithm, which incorporates a natural chromosome encoding format and specially designed genetic operators, is developed. An efficient unlocking method is constructed to address the unavoidable dead-lock phenomenon meanwhile maintaining the population randomness. Numerical simulations for different problem sizes and ammunition stocks are performed, and the proposed algorithm is compared with the Multi-objective Particle Swarm Optimization and the Multi-objective Grey Wolf Optimization, respectively, using different unlocking approaches. The simulation and comparison results demonstrate the practical value and effectiveness of the developed model and the proposed algorithm.

AAAI Conference 2023 Conference Paper

DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video

• Zhimeng Zhang
• Zhipeng Hu
• Wenjin Deng
• Changjie Fan
• Tangjie Lv
• Yu Ding

For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists of one deformation part and one inpainting part. In the first part, five reference facial images adaptively perform spatial deformation to create deformed feature maps encoding mouth shapes at each frame, in order to align with input driving audio and also the head poses of input source images. In the second part, to produce face visually dubbing, a feature decoder is responsible for adaptively incorporating mouth movements from the deformed feature maps and other attributes (i.e., head pose and upper facial expression) from the source feature maps together. Finally, DINet achieves face visually dubbing with rich textural details. We conduct qualitative and quantitative comparisons to validate our DINet on high-resolution videos. The experimental results show that our method outperforms state-of-the-art works.

AAAI Conference 2023 Conference Paper

FlowFace: Semantic Flow-Guided Shape-Aware Face Swapping

• Hao Zeng
• Wei Zhang
• Changjie Fan
• Tangjie Lv
• Suzhen Wang
• Zhimeng Zhang
• Bowen Ma
• Lincheng Li

In this work, we propose a semantic flow-guided two-stage framework for shape-aware face swapping, namely FlowFace. Unlike most previous methods that focus on transferring the source inner facial features but neglect facial contours, our FlowFace can transfer both of them to a target face, thus leading to more realistic face swapping. Concretely, our FlowFace consists of a face reshaping network and a face swapping network. The face reshaping network addresses the shape outline differences between the source and target faces. It first estimates a semantic flow (i.e. face shape differences) between the source and the target face, and then explicitly warps the target face shape with the estimated semantic flow. After reshaping, the face swapping network generates inner facial features that exhibit the identity of the source face. We employ a pre-trained face masked autoencoder (MAE) to extract facial features from both the source face and the target face. In contrast to previous methods that use identity embedding to preserve identity information, the features extracted by our encoder can better capture facial appearances and identity information. Then, we develop a cross-attention fusion module to adaptively fuse inner facial features from the source face with the target facial attributes, thus leading to better identity preservation. Extensive quantitative and qualitative experiments on in-the-wild faces demonstrate that our FlowFace outperforms the state-of-the-art significantly.

AAAI Conference 2023 Conference Paper

Multi-Scale Control Signal-Aware Transformer for Motion Synthesis without Phase

• Lintao Wang
• Kun Hu
• Lei Bai
• Yu Ding
• Wanli Ouyang
• Zhiyong Wang

Synthesizing controllable motion for a character using deep learning has been a promising approach due to its potential to learn a compact model without laborious feature engineering. To produce dynamic motion from weak control signals such as desired paths, existing methods often require auxiliary information such as phases for alleviating motion ambiguity, which limits their generalisation capability. As past poses often contain useful auxiliary hints, in this paper, we propose a task-agnostic deep learning method, namely Multi-scale Control Signal-aware Transformer (MCS-T), with an attention based encoder-decoder architecture to discover the auxiliary information implicitly for synthesizing controllable motion without explicitly requiring auxiliary information such as phase. Specifically, an encoder is devised to adaptively formulate the motion patterns of a character's past poses with multi-scale skeletons, and a decoder driven by control signals to further synthesize and predict the character's state by paying context-specialised attention to the encoded past motion patterns. As a result, it helps alleviate the issues of low responsiveness and slow transition which often happen in conventional methods not using auxiliary information. Both qualitative and quantitative experimental results on an existing biped locomotion dataset, which involves diverse types of motion transitions, demonstrate the effectiveness of our method. In particular, MCS-T is able to successfully generate motions comparable to those generated by the methods using auxiliary information.

AAAI Conference 2023 Conference Paper

StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

• Yifeng Ma
• Suzhen Wang
• Zhipeng Hu
• Changjie Fan
• Tangjie Lv
• Yu Ding
• Zhidong Deng
• Xin Yu

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable head motions, they still cannot generate diverse speaking styles in the final talking head videos. To tackle this problem, we propose a one-shot style-controllable talking face generation framework. In a nutshell, we aim to attain a speaking style from an arbitrary reference speaking video and then drive the one-shot portrait to speak with the reference speaking style and another piece of audio. Specifically, we first develop a style encoder to extract dynamic facial motion patterns of a style reference video and then encode them into a style code. Afterward, we introduce a style-controllable decoder to synthesize stylized facial animations from the speech content and style code. In order to integrate the reference speaking style into generated videos, we design a style-aware adaptive transformer, which enables the encoded style code to adjust the weights of the feed-forward layers accordingly. Thanks to the style-aware adaptation mechanism, the reference speaking style can be better embedded into synthesized videos during decoding. Extensive experiments demonstrate that our method is capable of generating talking head videos with diverse speaking styles from only one portrait image and an audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.

NeurIPS Conference 2022 Conference Paper

Domain Generalization by Learning and Removing Domain-specific Features

• Yu Ding
• Lei Wang
• Bin Liang
• Shuming Liang
• Yang Wang
• Fang Chen

Deep Neural Networks (DNNs) suffer from domain shift when the test dataset follows a distribution different from the training dataset. Domain generalization aims to tackle this issue by learning a model that can generalize to unseen domains. In this paper, we propose a new approach that aims to explicitly remove domain-specific features for domain generalization. Following this approach, we propose a novel framework called Learning and Removing Domain-specific features for Generalization (LRDG) that learns a domain-invariant model by tactically removing domain-specific features from the input images. Specifically, we design a classifier to effectively learn the domain-specific features for each source domain, respectively. We then develop an encoder-decoder network to map each input image into a new image space where the learned domain-specific features are removed. With the images output by the encoder-decoder network, another classifier is designed to learn the domain-invariant features to conduct image classification. Extensive experiments demonstrate that our framework achieves superior performance compared with state-of-the-art methods.

IJCAI Conference 2022 Conference Paper

MMT: Multi-way Multi-modal Transformer for Multimodal Learning

• Jiajia Tang
• Kang Li
• Ming Hou
• Xuanyu Jin
• Wanzeng Kong
• Yu Ding
• Qibin Zhao

The heart of multimodal learning research lies the challenge of effectively exploiting fusion representations among multiple modalities. However, existing two-way cross-modality unidirectional attention could only exploit the intermodal interactions from one source to one target modality. This indeed fails to unleash the complete expressive power of multimodal fusion with restricted number of modalities and fixed interactive direction. In this work, the multiway multimodal transformer (MMT) is proposed to simultaneously explore multiway multimodal intercorrelations for each modality via single block rather than multiple stacked cross-modality blocks. The core idea of MMT is the multiway multimodal attention, where the multiple modalities are leveraged to compute the multiway attention tensor. This naturally benefits us to exploit comprehensive many-to-many multimodal interactive paths. Specifically, the multiway tensor is comprised of multiple interconnected modality-aware core tensors that consist of the intramodal interactions. Additionally, the tensor contraction operation is utilized to investigate intermodal dependencies between distinct core tensors. Essentially, our tensor-based multiway structure allows for easily extending MMT to the case associated with an arbitrary number of modalities. Taking MMT as the basis, the hierarchical network is further established to recursively transmit the low-level multiway multimodal interactions to high-level ones. The experiments demonstrate that MMT can achieve state-of-the-art or comparable performance.

AAAI Conference 2022 Conference Paper

Multi-Dimensional Prediction of Guild Health in Online Games: A Stability-Aware Multi-Task Learning Approach

• Chuang Zhao
• Hongke Zhao
• Runze Wu
• Qilin Deng
• Yu Ding
• Jianrong Tao
• Changjie Fan

Guild is the most important long-term virtual community and emotional bond in massively multiplayer online roleplaying games (MMORPGs). It matters a lot to the player retention and game ecology how the guilds are going, e. g. , healthy or not. The main challenge now is to characterize and predict the guild health in a quantitative, dynamic, and multi-dimensional manner based on complicated multimedia data streams. To this end, we propose a novel framework, namely Stability-Aware Multi-task Learning Approach (SAMLA) to address these challenges. Specifically, different media-specific modules are designed to extract information from multiple media types of tabular data, time series characteristics, and heterogeneous graphs. To capture the dynamics of guild health, we introduce a representation encoder to provide a time-series view of multi-media data that is used for task prediction. Inspired by well-received theories on organization management, we delicately define five specific and quantitative dimensions of guild health and make parallel predictions based on a multi-task approach. Besides, we devise a novel auxiliary task, i. e. , the guild stability, to boost the performance of the guild health prediction task. Extensive experiments on a real-world large-scale MMORPG dataset verify that our proposed method outperforms the state-of-the-art methods in the task of organizational health characterization and prediction. Moreover, our work has been practically deployed in online MMORPG, and case studies clearly illustrate the significant value.

AAMAS Conference 2022 Conference Paper

Multimodal Reinforcement Learning with Effective State Representation Learning

• Jinming Ma
• Yingfeng Chen
• Feng Wu
• Xianpeng Ji
• Yu Ding

Many real-world applications require an agent to make robust and deliberate decisions with multimodal information (e. g. , robots with multi-sensory inputs). However, it is very challenging to train the agent via reinforcement learning (RL) due to the heterogeneity and dynamic importance of different modalities. Specifically, we observe that these issues make conventional RL methods difficult to learn a useful state representation in the end-to-end training with multimodal information. To address this, we propose a novel multimodal RL approach that can do multimodal alignment and importance enhancement according to their similarity and importance in terms of RL tasks respectively. By doing so, we are able to learn an effective state representation and consequentially improve the RL training process. We test our approach on several multimodal RL domains, showing that it outperforms state-of-the-art methods in terms of learning speed and policy quality.

AAAI Conference 2022 Conference Paper

One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning

• Suzhen Wang
• Lincheng Li
• Yu Ding
• Xin Yu

Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons. However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those methods struggle to learn a consistent speech style from different speakers. We observe that it would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements. Hence, we propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker and then transferring audio-driven motion fields to a reference image. Specifically, we develop an Audio-Visual Correlation Transformer (AVCT) that aims to infer talking motions represented by keypoint based dense motion fields from an input audio. In particular, considering audio may come from different identities in deployment, we incorporate phonemes to represent audio signals. In this manner, our AVCT can inherently generalize to audio spoken by other identities. Moreover, as face keypoints are used to represent speakers, AVCT is agnostic against appearances of the training speaker, and thus allows us to manipulate face images of different identities readily. Considering different face shapes lead to different motions, a motion field transfer module is exploited to reduce the audio-driven dense motion field gap between the training identity and the one-shot reference. Once we obtained the dense motion field of the reference image, we employ an image renderer to generate its talking face videos from an audio clip. Thanks to our learned consistent speaking style, our method generates authentic mouth shapes and vivid movements. Extensive experiments demonstrate that our synthesized videos outperform the state-of-the-art in terms of visual quality and lip-sync.

IJCAI Conference 2021 Conference Paper

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

• Suzhen Wang
• Lincheng Li
• Yu Ding
• Changjie Fan
• Xin Yu

We propose an audio-driven talking-head method to generate photo-realistic talking-head videos from a single reference image. In this work, we tackle two key challenges: (i) producing natural head motions that match speech prosody, and (ii)} maintaining the appearance of a speaker in a large head motion while stabilizing the non-face regions. We first design a head pose predictor by modeling rigid 6D head movements with a motion-aware recurrent neural network (RNN). In this way, the predicted head poses act as the low-frequency holistic movements of a talking head, thus allowing our latter network to focus on detailed facial movement generation. To depict the entire image motions arising from audio, we exploit a keypoint based dense motion field representation. Then, we develop a motion field generator to produce the dense motion fields from input audio, head poses, and a reference image. As this keypoint based representation models the motions of facial regions, head, and backgrounds integrally, our method can better constrain the spatial and temporal consistency of the generated videos. Finally, an image generation network is employed to render photo-realistic talking-head videos from the estimated keypoint based motion fields and the input reference image. Extensive experiments demonstrate that our method produces videos with plausible head motions, synchronized facial expressions, and stable backgrounds and outperforms the state-of-the-art.

JMLR Journal 2021 Journal Article

Neighborhood Structure Assisted Non-negative Matrix Factorization and Its Application in Unsupervised Point-wise Anomaly Detection

• Imtiaz Ahmed
• Xia Ben Hu
• Mithun P. Acharya
• Yu Ding

Dimensionality reduction is considered as an important step for ensuring competitive performance in unsupervised learning such as anomaly detection. Non-negative matrix factorization (NMF) is a widely used method to accomplish this goal. But NMF do not have the provision to include the neighborhood structure information and, as a result, may fail to provide satisfactory performance in presence of nonlinear manifold structure. To address this shortcoming, we propose to consider the neighborhood structural similarity information within the NMF framework and do so by modeling the data through a minimum spanning tree. We label the resulting method as the neighborhood structure-assisted NMF. We further develop both offline and online algorithms for implementing the proposed method. Empirical comparisons using twenty benchmark data sets as well as an industrial data set extracted from a hydropower plant demonstrate the superiority of the neighborhood structure-assisted NMF. Looking closer into the formulation and properties of the proposed NMF method and comparing it with several NMF variants reveal that inclusion of the MST-based neighborhood structure plays a key role in attaining the enhanced performance in anomaly detection. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2021. ( edit, beta )

AAAI Conference 2021 Conference Paper

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

• Lincheng Li
• Suzhen Wang
• Zhimeng Zhang
• Yu Ding
• Yixing Zheng
• Xin Yu
• Changjie Fan

In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailored for different individuals. It takes the animation parameters as input and exploits an attention mask to manipulate facial expression changes for the input individuals. Furthermore, to better establish authentic correspondences between visual motions (i. e. , facial expression changes and head movements) and audios, we leverage a high-accuracy motion capture dataset instead of relying on long videos of specific individuals. After attaining the visual and audio correspondences, we can effectively train our network in an end-to-end fashion. Extensive experiments on qualitative and quantitative results demonstrate that our algorithm achieves high-quality photorealistic talking-head videos including various facial expressions and head motions according to speech rhythms and outperforms the state-of-the-art.

JMLR Journal 2015 Journal Article

Absent Data Generating Classifier for Imbalanced Class Sizes

• Arash Pourhabib
• Bani K. Mallick
• Yu Ding

We propose an algorithm for two-class classification problems when the training data are imbalanced. This means the number of training instances in one of the classes is so low that the conventional classification algorithms become ineffective in detecting the minority class. We present a modification of the kernel Fisher discriminant analysis such that the imbalanced nature of the problem is explicitly addressed in the new algorithm formulation. The new algorithm exploits the properties of the existing minority points to learn the effects of other minority data points, had they actually existed. The algorithm proceeds iteratively by employing the learned properties and conditional sampling in such a way that it generates sufficient artificial data points for the minority set, thus enhancing the detection probability of the minority class. Implementing the proposed method on a number of simulated and real data sets, we show that our proposed method performs competitively compared to a set of alternative state-of-the-art imbalanced classification algorithms. [abs] [ pdf ][ bib ] &copy JMLR 2015. ( edit, beta )

AAAI Conference 2015 Conference Paper

LOL — Laugh Out Loud

• Florian Pecune
• Beatrice Biancardi
• Yu Ding
• Catherine Pelachaud
• Maurizio Mancini
• Giovanna Varni
• Antonio Camurri
• Gualtiero Volpe

JMLR Journal 2012 Journal Article

GPLP: A Local and Parallel Computation Toolbox for Gaussian Process Regression

• Chiwoo Park
• Jianhua Z. Huang
• Yu Ding

This paper presents the Getting-started style documentation for the local and parallel computation toolbox for Gaussian process regression (GPLP), an open source software package written in Matlab (but also compatible with Octave). The working environment and the usage of the software package will be presented in this paper. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2012. ( edit, beta )

JMLR Journal 2011 Journal Article

Domain Decomposition Approach for Fast Gaussian Process Regression of Large Spatial Data Sets

• Chiwoo Park
• Jianhua Z. Huang
• Yu Ding

Gaussian process regression is a flexible and powerful tool for machine learning, but the high computational complexity hinders its broader applications. In this paper, we propose a new approach for fast computation of Gaussian process regression with a focus on large spatial data sets. The approach decomposes the domain of a regression function into small subdomains and infers a local piece of the regression function for each subdomain. We explicitly address the mismatch problem of the local pieces on the boundaries of neighboring subdomains by imposing continuity constraints. The new approach has comparable or better computation complexity as other competing methods, but it is easier to be parallelized for faster computation. Moreover, the method can be adaptive to non-stationary features because of its local nature and, in particular, its use of different hyperparameters of the covariance function for different local regions. We illustrate application of the method and demonstrate its advantages over existing methods using two synthetic data sets and two real spatial data sets. [abs] [ pdf ][ bib ] &copy JMLR 2011. ( edit, beta )