Author name cluster

Chao Yao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

1 author row

EAAI Journal 2026 Journal Article

A novel multi-modal attentional collaborative learning framework with semantic enhancement for audio–visual question answering

Jie Yang
Miao Ma
Peng Wang
Yutong Li
Zhao Pei
Chao Yao
Longjiang Guo

The Audio–Visual Question Answering (AVQA) task aims to extract audio and visual cues from videos for answering the questions. The popular two-stage method, such as Progressive Spatio-Temporal Perception Network (PSTP-Net), first locates key segments in the audio–visual scene based on the question and then identifies the most relevant audio–visual regions. While this reduces cue redundancy, it overlooks the complementary role of rich cues, which is crucial for a comprehensive understanding of audio–visual content. In this paper, we propose a novel framework to start from the question itself, guide the entire multi-modal collaborative learning process, and conduct audio–visual question answering. This method includes a semantically enhanced strategy using Multi-modal Large Language Models (MLLMs) applied as an engineering solution, and a multi-modal attentional collaborative learning process, which is the core algorithmic innovation. Extensive experiments on the Music Audio–Visual Question Answering dataset (MUSIC-AVQA) and Music Audio–Visual Question Answering dataset version 2 (MUSIC-AVQA v2) demonstrate the effectiveness of our method. Compared to the PSTP-Net, our method reduces the number of training parameters by 61. 23% and Floating-point Operations (FLOPs) by 60. 83%, while achieving 2. 61 percentage-point improvement in accuracy. This indicates that our method effectively captures and aligns rich audio–visual cues, significantly enhancing reasoning efficiency. Our code will be publicly available soon.

Details DOI

NeurIPS Conference 2024 Conference Paper

SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution

Qi Tang
Yao Zhao
Meiqin Liu
Chao Yao

Diffusion-based Video Super-Resolution (VSR) is renowned for generating perceptually realistic videos, yet it grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach of pixel-level alignment is ineffective for diffusion-processed frames because of iterative disruptions. To overcome this, we introduce SeeClear--a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Condenser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centric Alignment Module (InCAM) utilizes video-clip-wise tokens to dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel-wise Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of-the-art diffusion-based VSR techniques.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Semantic Lens: Instance-Centric Semantic Alignment for Video Super-resolution

Qi Tang
Yao Zhao
Meiqin Liu
Jian Jin
Chao Yao

As a critical clue of video super-resolution (VSR), inter-frame alignment significantly impacts overall performance. However, accurate pixel-level alignment is a challenging task due to the intricate motion interweaving in the video. In response to this issue, we introduce a novel paradigm for VSR named Semantic Lens, predicated on semantic priors drawn from degraded videos. Specifically, video is modeled as instances, events, and scenes via a Semantic Extractor. Those semantics assist the Pixel Enhancer in understanding the recovered contents and generating more realistic visual results. The distilled global semantics embody the scene information of each frame, while the instance-specific semantics assemble the spatial-temporal contexts related to each instance. Furthermore, we devise a Semantics-Powered Attention Cross-Embedding (SPACE) block to bridge the pixel-level features with semantic knowledge, composed of a Global Perspective Shifter (GPS) and an Instance-Specific Semantic Embedding Encoder (ISEE). Concretely, the GPS module generates pairs of affine transformation parameters for pixel-level feature modulation conditioned on global semantics. After that the ISEE module harnesses the attention mechanism to align the adjacent frames in the instance-centric semantic space. In addition, we incorporate a simple yet effective pre-alignment module to alleviate the difficulty of model training. Extensive experiments demonstrate the superiority of our model over existing state-of-the-art VSR methods.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics

Xueyuan Yang
Chao Yao
Xiaojuan Ban

Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMU data is inherently fraught with ambiguity, a consequence of numerous identical IMU readings corresponding to different poses. In this paper, we explore the spatial importance of sparse sensors, supervised by text that describes specific actions. Specifically, uncertainty is introduced to derive weighted features for each IMU. We also design a Hierarchical Temporal Transformer (HTT) and apply contrastive learning to achieve precise temporal and feature alignment of sensor data with textual semantics. Experimental results demonstrate our proposed approach achieves significant improvements in multiple metrics compared to existing methods. Notably, with textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Who Looks like Me: Semantic Routed Image Harmonization

Jinsheng Sun
Chao Yao
Xiaokun Wang
Yu Guo
Yalan Zhang
Xiaojuan Ban

Image harmonization, aiming to seamlessly blend extraneous foreground objects with background images, is a promising and challenging task. Ensuring a synthetic image appears realistic requires maintaining consistency in visual characteristics, such as texture and style, across global and semantic regions. In this paper, We approach image harmonization as a semantic routed style transfer problem, and propose an imageharmonization model by routing semantic similarity explicitly to enhance the consistency of appearance characteristics. To refine calculate the similarity between the composed foreground and background instance, we propose an InstanceSimilarity Evaluation Module(ISEM). To harness analogous semantic information effectively, we further introduceStyle Transfer Block(STB) to establish fine-grained foreground-background semantic correlation. Our method has achieved excellent experimental results on existing datasets and our model outperforms the state-of-the-art by a margin of 0. 45 dB on iHarmony4 dataset.

PDF Details DOI