Author name cluster

Yao Qian

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Leying Zhang
Yao Qian
Xiaofei Wang
Manthan Thakker
Dongmei Wang
Jianwei Yu
Haibin Wu
Yuxuan Hu

Generating natural-sounding, multi-speaker dialogue is crucial for applications such as podcast creation, virtual agents, and multimedia content generation. However, existing systems struggle to maintain speaker consistency, model overlapping speech, and synthesize coherent conversations efficiently. In this paper, we introduce CoVoMix2, a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. CoVoMix2 directly predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model, eliminating the reliance on intermediate token representations. To better capture realistic conversational dynamics, we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies. Our approach achieves state-of-the-art performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed. Notably, CoVoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and precise timing control, demonstrating strong generalizability to real-world speech generation scenarios. Audio samples are available at https: //www. microsoft. com/en-us/research/project/covomix/covomix2.

PDF Details

EAAI Journal 2025 Journal Article

Transformer architecture with illumination aware mechanisms for low-light image enhancement via Retinex decomposition

Zixuan Wang
Gang Liu
Hanlin Xu
Yao Qian
Rui Chang
Durga Prasad Bavirisetti

Enhancing low-light images is a complex task that involves not only restoring brightness but also preserving color fidelity and reducing noise interference. In this paper, we propose a novel Retinex-based Transformer Model with Illumination Aware Mechanisms (TIMRetinex-Net), which achieves physically interpretable modeling through a decomposition network guided by Retinex theory. To adapt to light variations in different regions, we randomly apply gamma transformations to several subregions of the illumination component and use a Color Estimation Module to capture the color global distribution of the natural scene in the reflection component. By modeling the color global distribution and repairing the degraded regions collaboratively, we alleviate the issue of being highly sensitive to data usage during training and improve the model’s ability to handle unknown scenes. The Illumination and Reflection Adjustment Transformer Network (IRAT-Net) produces enhanced images, achieving a balanced enhancement of detail and color. In addition, IRAT-Net incorporates an attention mechanism into the feature extraction layer and introduces the Illumination-Guided Information Aggregation Module to adaptively estimate lighting conditions. In the field of image processing, our method based on artificial intelligence was evaluated on five datasets and compared with twelve state-of-the-art methods. The results demonstrated strong alignment with the ground truth, with our method achieving superior performance in both subjective and objective assessments.

Details DOI

NeurIPS Conference 2024 Conference Paper

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

Leying Zhang
Yao Qian
Long Zhou
Shujie Liu
Dongmei Wang
Xiaofei Wang
Midia Yousefi
Yanmin Qian

Recent advancements in zero-shot text-to-speech (TTS) modeling have led to significant strides in generating high-fidelity and diverse speech. However, dialogue generation, along with achieving human-like naturalness in speech, continues to be a challenge. In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation. CoVoMix first converts dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers. These token streams are then fed into a flow-matching based acoustic model to generate mixed mel-spectrograms. Finally, the speech waveforms are produced using a HiFi-GAN model. Furthermore, we devise a comprehensive set of metrics for measuring the effectiveness of dialogue modeling and generation. Our experimental results show that CoVoMix can generate dialogues that are not only human-like in their naturalness and coherence but also involve multiple talkers engaging in multiple rounds of conversation. This is exemplified by instances generated in a single channel where one speaker's utterance is seamlessly mixed with another's interjections or laughter, indicating the latter's role as an attentive listener. Audio samples are enclosed in the supplementary.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

Chenyang Le
Yao Qian
Dongmei Wang
Long Zhou
Shujie Liu
Xiaofei Wang
Midia Yousefi
Yanmin Qian

There is a rising interest and trend in research towards directly translating speech from one language to another, known as end-to-end speech-to-speech translation. However, most end-to-end models struggle to outperform cascade models, i. e. , a pipeline framework by concatenating speech recognition, machine translation and text-to-speech models. The primary challenges stem from the inherent complexities involved in direct translation tasks and the scarcity of data. In this study, we introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion yet facilitates end-to-end inference through joint probability. Furthermore, we propose two separated encoders to preserve the speaker’s voice characteristics and isochrony from the source speech during the translation process, making it highly suitable for scenarios such as video dubbing. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation

Chenyang Le
Yao Qian
Long Zhou
Shujie Liu
Yanmin Qian
Michael Zeng
Xuedong Huang

Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pre-trained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31. 5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.

PDF Details

AAAI Conference 2023 Conference Paper

i-Code: An Integrative and Composable Multimodal Learning Framework

Ziyi Yang
Yuwei Fang
Chenguang Zhu
Reid Pryzant
DongDong Chen
Yu Shi
Yichong Xu
Yao Qian

Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code, a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. In this framework, data from each modality are first given to pretrained single-modality encoders. The encoder outputs are then integrated with a multimodal fusion network, which uses novel merge- and co-attention mechanisms to effectively combine information from the different modalities. The entire system is pretrained end-to-end with new objectives including masked modality unit modeling and cross-modality contrastive learning. Unlike previous research using only video for pretraining, the i-Code framework can dynamically process single, dual, and triple-modality data during training and inference, flexibly projecting different combinations of modalities into a single representation space. Experimental results demonstrate how i-Code can outperform state-of-the-art techniques on five multimodal understanding tasks and single-modality benchmarks, improving by as much as 11% and demonstrating the power of integrative multimodal pretraining.

PDF Details DOI

ICML Conference 2021 Conference Paper

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

Chengyi Wang 0002
Yu Wu 0012
Yao Qian
Ken'ichi Kumatani
Shujie Liu 0001
Furu Wei
Michael Zeng 0001
Xuedong Huang 0001

In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both labeled and unlabeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture information more correlated with phonetic structures and improve the generalization across languages and domains. We evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The results show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech recognition by a maximum of 13. 4% and 26. 9% relative phone error rate reductions respectively (averaged over all testing languages). The transferability of UniSpeech is also verified on a domain-shift speech recognition task, i. e. , a relative word error rate reduction of 6% against the previous approach.

Details