Author name cluster

Shan Yang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

AAAI Conference 2026 Conference Paper

UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation

Jinting Wang
Shan Yang
Chenxing Li
Dong Yu
Li Liu

Cued Speech (CS) enhances lipreading via hand coding, offering visual phonemic cues that support precise speech perception for the hearing-impaired. The task of CS Video-to-Speech generation (CSV2S) aims to convert CS videos into intelligible speech signals. Most existing research focuses on CS Recognition (CSR), which transcribes video content into text. Consequently, a common solution for CSV2S is to integrate CSR with a text-to-speech (TTS) system. However, this pipeline relies on text as an intermediate medium, which may lead to error propagation and temporal misalignment between speech and CS video dynamics. In contrast, directly generating audio speech from CS video (direct CSV2S) often suffer from the inherent multimodal complexity and the limited availability of CS data. To address these challenges, we propose UniCUE, the first unified framework for CSV2S that directly generates speech from CS videos without relying on intermediate text. The core innovation of UniCUE lies in integrating a understanding task (CSR) that provides fine-grained CS visual-semantic cues to to guide the speech generation. Specifically, UniCUE incorporates a pose-aware visual processor, a semantic alignment pool that enables precise visual–semantic mapping, and a VisioPhonetic adapter to bridge the understanding and generation tasks within a unified architecture. To support this framework, we construct UniCUE-HI, a large-scale Mandarin CS dataset containing 11,282 videos from 14 cuers, including both hearing-impaired and normal-hearing individuals. Extensive experiments conducted on this dataset demonstrate that UniCUE achieves state-of-the-art (SOTA) performance across multiple evaluation metrics.

PDF Details DOI

YNIMG Journal 2025 Journal Article

Glucocorticoid receptor gene (NR3C1) methylation, childhood maltreatment, multilevel reward responsiveness and depressive and anxiety symptoms: A neuroimaging epigenetic study

Yajing Xu
Shan Yang
Cong Cao

BACKGROUND: Although epigenomic and environment interactions (Epigenome × Environment; Epi × E) might constitute a novel mechanism underlying reward processing, direct evidence is still scarce. We conducted the first longitudinal study to investigate the extent to which DNA methylation of a stress-related gene-NR3C1-interacts with childhood maltreatment in association with young adult reward responsiveness (RR) and the downstream risk of depressive (anhedonia dimension in particular) and anxiety symptoms. METHOD: = 21.08 ± 1.91 years; 59.4% females) were followed in two waves. Reward positivity (RewP) and its time‒frequency components were elicited via a classic monetary reward task. Cytosine methylation in the promoter exon 1F of NR3C1 (NR3C1-1F) was sequenced via buccal cells. Childhood maltreatment, self-reported RR and depressive and anxiety symptoms were assessed via questionnaires. RESULTS: NR3C1-1F methylation significantly interacted with childhood maltreatment on RewP but not the delta and theta components or self-reported RR. The severity and exposure number of childhood maltreatment were negatively associated with RewP among individuals with heightened NR3C1-1F methylation but positively associated with RewP among individuals with blunted NR3C1-1F methylation, demonstrating a "goodness-of-fit" interaction. This interaction was specifically linked with anhedonia dimension but not with total scores of depressive or anxiety symptoms. CONCLUSIONS: The current findings provide preliminary evidence for an Epi × E interaction underlying reward processing, highlight cross-level analyses of electrophysiological signals and advance knowledge of the biological foundation of stress-induced reward function and relevant symptoms. However, caution should be paid to the generalizability of these findings in high-risk clinical samples given the high-functioning characteristic of the present sample.

Details DOI

ICRA Conference 2024 Conference Paper

EffLoc: Lightweight Vision Transformer for Efficient 6-DOF Camera Relocalization

Zhendong Xiao
Changhao Chen
Shan Yang
Wu Wei 0001

Camera relocalization is pivotal in computer vision, with applications in AR, drones, robotics, and autonomous driving. It estimates 3D camera position and orientation (6-DoF) from images. Unlike traditional methods like SLAM, recent strides use deep learning for direct end-to-end pose estimation. We propose EffLoc, a novel efficient Vision Transformer for single-image camera relocalization. EffLoc’s hierarchical layout, memory-bound self-attention, and feed-forward layers boost memory efficiency and inter-channel communication. Our introduced sequential group attention (SGA) module enhances computational efficiency by diversifying input features, reducing redundancy, and expanding model capacity. EffLoc excels in efficiency and accuracy, outperforming prior methods, such as AtLoc and MapNet. It thrives on large-scale outdoor car-driving scenario, ensuring simplicity, end-to-end trainability, and eliminating handcrafted loss functions.

Details

AAAI Conference 2024 Conference Paper

ICAR: Image-Based Complementary Auto Reasoning

Xijun Wang
Anqi Liang
Junbang Liang
Ming Lin
Yu Lou
Shan Yang

Scene-aware Complementary Item Retrieval (CIR) is a challenging task which requires to generate a set of compatible items across domains. Due to the subjectivity, it is difficult to set up a rigorous standard for both data collection and learning objectives. To address this challenging task, we propose a visual compatibility concept, composed of similarity (resembling in color, geometry, texture, and etc.) and complementarity (different items like table vs chair completing a group). Based on this notion, we propose a compatibility learning framework, a category-aware Flexible Bidirectional Transformer (FBT), for visual ``scene-based set compatibility reasoning'' with the cross-domain visual similarity input and auto-regressive complementary item generation. We introduce a ``Flexible Bidirectional Transformer (FBT),'' consisting of an encoder with flexible masking, a category prediction arm, and an auto-regressive visual embedding prediction arm. And the inputs for FBT are cross-domain visual similarity invariant embeddings, making this framework quite generalizable. Furthermore, our proposed FBT model learns the inter-object compatibility from a large set of scene images in a self-supervised way. Compared with the SOTA methods, this approach achieves up to 5.3% and 9.6% in FITB score and 22.3% and 31.8% SFID improvement on fashion and furniture, respectively.

PDF Details DOI

AAAI Conference 2023 Conference Paper

UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis

Yi Lei
Shan Yang
Xinsheng Wang
Qicong Xie
Jixun Yao
Lei Xie
Dan Su

Text-to-speech (TTS) and singing voice synthesis (SVS) aim at generating high-quality speaking and singing voice according to textual input and music scores, respectively. Unifying TTS and SVS into a single system is crucial to the applications requiring both of them. Existing methods usually suffer from some limitations, which rely on either both singing and speaking data from the same person or cascaded models of multiple tasks. To address these problems, a simplified elegant framework for TTS and SVS, named UniSyn, is proposed in this paper. It is an end-to-end unified model that can make a voice speak and sing with only singing or speaking data from this person. To be specific, a multi-conditional variational autoencoder (MC-VAE), which constructs two independent latent sub-spaces with the speaker- and style-related (i.e. speak or sing) conditions for flexible control, is proposed in UniSyn. Moreover, supervised guided-VAE and timbre perturbation with the Wasserstein distance constraint are leveraged to further disentangle the speaker timbre and style. Experiments conducted on two speakers and two singers demonstrate that UniSyn can generate natural speaking and singing voice without corresponding training data. The proposed approach outperforms the state-of-the-art end-to-end voice generation work, which proves the effectiveness and advantages of UniSyn.

PDF Details DOI

NeurIPS Conference 2021 Conference Paper

Attention Bottlenecks for Multimodal Fusion

Arsha Nagrani
Shan Yang
Anurag Arnab
Aren Jansen
Cordelia Schmid
Chen Sun

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks. A common approach for building multimodal models is to simply combine multiple of these modality-specific architectures using late-stage fusion of final representations or predictions ('late-fusion'). Instead, we introduce a novel transformer based architecture that uses 'attention bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, these bottlenecks force information between different modalities to pass through a small number of '`bottleneck' latent units, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

PDF Details

YNICL Journal 2018 Journal Article

Voxel-wise deviations from healthy aging for the detection of region-specific atrophy

Stefan Klöppel
Shan Yang
Elias Kellner
Marco Reisert
Bernhard Heimbach
Horst Urbach
Jennifer Linn
Stefan Weidauer

The identification of pathological atrophy in MRI scans requires specialized training, which is scarce outside dedicated centers. We sought to investigate the clinical usefulness of computer-generated representations of local grey matter (GM) loss or increased volume of cerebral fluids (CSF) as normalized deviations (z-scores) from healthy aging to either aid human visual readings or directly detect pathological atrophy. Two experienced neuroradiologists rated atrophy in 30 patients with Alzheimer's disease (AD), 30 patients with frontotemporal dementia (FTD), 30 with dementia due to Lewy-body disease (LBD) and 30 healthy controls (HC) on a three-point scale in 10 anatomical regions as reference gold standard. Seven raters, varying in their experience with MRI diagnostics rated all cases on the same scale once with and once without computer-generated volume deviation maps that were overlaid on anatomical slices. In addition, we investigated the predictive value of the computer generated deviation maps on their own for the detection of atrophy as identified by the gold standard raters. Inter and intra-rater agreements of the two gold standard raters were substantial (Cohen's kappa κ > 0.62). The intra-rater agreement of the other raters ranged from fair (κ = 0.37) to substantial (κ = 0.72) and improved on average by 0.13 (0.57 < κ < 0.87) when volume deviation maps were displayed. The seven other raters showed good agreement with the gold standard in regions including the hippocampus but agreement was substantially lower in e.g. the parietal cortex and did not improve with the display of atrophy scores. Rating speed increased over the course of the study and irrespective of the presentation of voxel-wise deviations. Automatically detected large deviations of local volume were consistently associated with gold standard atrophy reading as shown by an area under the receiver operator characteristic of up to 0.95 for the hippocampus region. When applying these test characteristics to prevalences typically found in a memory clinic, we observed a positive or negative predictive value close to or above 0.9 in the hippocampus for almost all of the expected cases. The volume deviation maps derived from CSF volume increase were generally better in detecting atrophy. Our study demonstrates an agreement of visual ratings among non-experts not further increased by displaying, region-specific deviations of volume. The high predictive value of computer generated local deviations independent from human interaction and the consistent advantages of CSF-over GM-based estimations should be considered in the development of diagnostic tools and indicate clinical utility well beyond aiding visual assessments.

Details DOI

ICRA Conference 2016 Conference Paper

Bayesian estimation of non-rigid mechanical parameters using temporal sequences of deformation samples

Shan Yang
Ming Lin 0003

Material property has great importance in medical robotics. The mechanical properties of the human soft tissue, are important to characterize the tissue deformation of each patient. The (recovered) elasticity parameters can assist surgeons to perform better pre-op surgical planning and enable medical robots to carry out personalized surgical procedures. In this paper, we present a novel algorithm on mechanical-property estimation from a temporal sequence of deformation samples. It does not require an external force-application measurement device or landmark-based displacement tracking. We test our approach on the reconstruction the Young's modulus of a human heart and further validate the results derived from videos using known parameters of tennis and foam balls.

Details