Author name cluster

Guoying Zhao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

13 papers

1 author row

IS Journal 2026 Journal Article

Evaluating the Adversarial Robustness of Vision–Language Models for Facial Expression Recognition

Hui Kuurila-Zhang
Haoyu Chen
Guoying Zhao

Facial expression recognition (FER) using vision–language models (VLMs) shows strong performance, but their robustness under adversarial conditions is underexplored. Noting that most VLMs rely on CLIP-style vision encoders vulnerable to gradient-based perturbations, we study how attacks on a CLIP encoder affect downstream recognition. We test zero-shot classifiers (CLIP, EVA-CLIP, Exp-CLIP) and generative VLMs (BLIP2, LLaVA, Qwen3-VL) performing FER via visual question answering. Using a CLIP surrogate for gradient-based attacks, we evaluate on AffectNet, RAF-DB, and FERPlus. Key findings include: 1) zero-shot classifiers are highly fragile, 2) attacks transfer only between models sharing the same vision encoder, and 3) generative VLMs are more robust than CLIP variants despite task-agnostic training. These results identify the vision encoder as the main bottleneck and highlight the need for robustness-focused design in future FER systems.

Details DOI

JBHI Journal 2026 Journal Article

ResoPhys: Unsupervised Plug-and-Play Remote Physiological Measurement via Facial Videos of Arbitrary Resolution

Zhongtian He
Shuyang Chu
Xuqi Li
Zhengdong Jiang
Jian Xu
Guoying Zhao
Jingang Shi

Remote photoplethysmography (rPPG) is a non-contact method that detects blood volume changes in facial tissues from video. The non-invasiveness of rPPG makes it promising for applications in remote health monitoring and telemedicine. However, its real-world application is hindered by a fundamental challenge. Existing models are typically designed for high-resolution, fixed-size inputs, making them ill-suited for the arbitrary-resolution videos commonly encountered in practical scenarios due to dynamic camera-to-subject distances. To address this challenge, we propose ResoPhys, an unsupervised plug-and-play rPPG measurement method designed for facial videos of arbitrary resolution. This method first generates video pairs via random scaling and then employs specialized modules for arbitrary-resolution feature extraction and upsampling to analyze the resulting multi-scale features. The framework is optimized via an unsupervised contrastive learning approach using our proposed multi-resolution contrastive loss. To validate its performance across a spectrum of resolutions, we evaluated ResoPhys on several public datasets. The results demonstrate the superiority of our method over previous unsupervised approaches, exhibiting particular strength in challenging low-resolution scenarios, which underscores its robustness to resolution changes. Crucially, ResoPhys acts as a universal front-end that decouples resolution handling from signal extraction, empowering existing rPPG networks for effective deployment in arbitrary-resolution conditions. The source code is available at: https://github.com/HeZhongTian-xjtu/ResoPhys.

Details DOI

JBHI Journal 2025 Journal Article

CodePhys: Robust Video-Based Remote Physiological Measurement Through Latent Codebook Querying

Shuyang Chu
Menghan Xia
Mengyao Yuan
Xin Liu
Tapio Seppänen
Guoying Zhao
Jingang Shi

Remote photoplethysmography (rPPG) aims to measure non-contact physiological signals from facial videos, which has shown great potential in many applications. Most existing methods directly extract video-based rPPG features by designing neural networks for heart rate estimation. Although they can achieve acceptable results, the recovery of rPPG signal faces intractable challenges when interference from real-world scenarios takes place on facial video. Specifically, facial videos are inevitably affected by non-physiological factors (e. g. , camera device noise, defocus, and motion blur), leading to the distortion of extracted rPPG signals. Recent rPPG extraction methods are easily affected by interference and degradation, resulting in noisy rPPG signals. In this paper, we propose a novel method named CodePhys, which innovatively treats rPPG measurement as a code query task in a noise-free proxy space (i. e. , codebook) constructed by ground-truth PPG signals. We consider noisy rPPG features as queries and generate high-fidelity rPPG features by matching them with noise-free PPG features from the codebook. Our approach also incorporates a spatial-aware encoder network with a spatial attention mechanism to highlight physiologically active areas and uses a distillation loss to reduce the influence of non-periodic visual interference. Experimental results on four benchmark datasets demonstrate that CodePhys outperforms state-of-the-art methods in both intra-dataset and cross-dataset settings.

Details DOI

AAAI Conference 2025 Conference Paper

FreeNet: Liberating Depth-Wise Separable Operations for Building Faster Mobile Vision Architectures

Hao Yu
Haoyu Chen
Wei Peng
Xu Cheng
Guoying Zhao

In the pursuit of efficient vision architectures, substantial efforts have been devoted to optimizing operator efficiency. Depth-wise separable operators, such as DWConv, are found cheap in both FLOPs and parameters. As a result, they are increasingly incorporated into efficient backbones, trading for deeper and wider architectures to enhance performance. However, separable operators are not really fast on devices due to the discontinuous memory access requirements. In this paper, we propose FreeNets, a family of simple and efficient backbones that free the separable operation to further accelerate the running speed. We introduce sparse sampling mixers (S2-Mixer) to supersede existing separable token mixers. The S2-Mixer samples multiple segments of partially continuous signals across spatial and channel dimensions for convolutional processing, achieving extremely fast on-device speed. The sparse sampling also enables S2-Mixer to capture long-range pixel relationships from dynamic receptive fields. Furthermore, we introduce a Shift Feed-Forward Network (ShiftFFN) as a faster alternative to existing channel mixers. It utilizes a shift neck architecture that aggregates global information to shift features, enabling faster channel mixing while incorporating global pixel information. Extensive experiments demonstrate that FreeNet offers a superior accuracy-efficiency tradeoff compared to the latest efficient models. On ImageNet-1k, FreeNet-S2 outperforms the StarNet-S4 by 0.4% in top-1 accuracy, while running around 40% faster on desktop GPU and 15% faster on Mobile GPU.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Differentiable Auxiliary Learning for Sketch Re-Identification

Xingyu Liu
Xu Cheng
Haoyu Chen
Hao Yu
Guoying Zhao

Sketch re-identification (Re-ID) seeks to match pedestrians' photos from surveillance videos with corresponding sketches. However, we observe that existing works still have two critical limitations: (i) cross- and intra-modality discrepancies hinder the extraction of modality-shared features, (ii) standard triplet loss fails to constrain latent feature distribution in each modality with inadequate samples. To overcome the above issues, we propose a differentiable auxiliary learning network (DALNet) to explore a robust auxiliary modality for Sketch Re-ID. Specifically, for (i) we construct an auxiliary modality by using a dynamic auxiliary generator (DAG) to bridge the gap between sketch and photo modalities. The auxiliary modality highlights the described person in photos to mitigate background clutter and learns sketch style through style refinement. Moreover, a modality interactive attention module (MIA) is presented to align the features and learn the invariant patterns of two modalities by auxiliary modality. To address (ii), we propose a multi-modality collaborative learning scheme (MMCL) to align the latent distribution of three modalities. An intra-modality circle loss in MMCL brings learned global and modality-shared features of the same identity closer in the case of insufficient samples within each modality. Extensive experiments verify the superior performance of our DALNet over the state-of-the-art methods for Sketch Re-ID, and the generalization in sketch-based image retrieval and sketch-photo face recognition tasks.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

LART: Neural Correspondence Learning with Latent Regularization Transformer for 3D Motion Transfer

Haoyu Chen
Hao Tang
Radu Timofte
Luc V Gool
Guoying Zhao

3D motion transfer aims at transferring the motion from a dynamic input sequence to a static 3D object and outputs an identical motion of the target with high-fidelity and realistic visual effects. In this work, we propose a novel 3D Transformer framework called LART for 3D motion transfer. With carefully-designed architectures, LART is able to implicitly learn the correspondence via a flexible geometry perception. Thus, unlike other existing methods, LART does not require any key point annotations or pre-defined correspondence between the motion source and target meshes and can also handle large-size full-detailed unseen 3D targets. Besides, we introduce a novel latent metric regularization on the Transformer for better motion generation. Our rationale lies in the observation that the decoded motions can be approximately expressed as linearly geometric distortion at the frame level. The metric preservation of motions could be translated to the formation of linear paths in the underlying latent space as a rigorous constraint to control the synthetic motions occurring in the construction of the latent space. The proposed LART shows a high learning efficiency with the need for a few samples from the AMASS dataset to generate motions with plausible visual effects. The experimental results verify the potential of our generative model in applications of motion transfer, content generation, temporal interpolation, and motion denoising. The code is made available: https: //github. com/mikecheninoulu/LART.

PDF Details

AAAI Conference 2022 Conference Paper

Geometry-Contrastive Transformer for Generalized 3D Pose Transfer

Haoyu Chen
Hao Tang
Zitong Yu
Nicu Sebe
Guoying Zhao

We present a customized 3D mesh Transformer model for the pose transfer task. As the 3D pose transfer essentially is a deformation procedure dependent on the given meshes, the intuition of this work is to perceive the geometric inconsistency between the given meshes with the powerful self-attention mechanism. Specifically, we propose a novel geometry-contrastive Transformer that has an efficient 3D structured perceiving ability to the global geometric inconsistencies across the given meshes. Moreover, locally, a simple yet efficient central geodesic contrastive loss is further proposed to improve the regional geometric-inconsistency learning. At last, we present a latent isometric regularization module together with a novel semi-synthesized dataset for the cross-dataset 3D pose transfer task towards unknown spaces. The massive experimental results prove the efficacy of our approach by showing state-of-the-art quantitative performances on SMPL-NPT, FAUST and our new proposed dataset SMG- 3D datasets, as well as promising qualitative results on MGcloth and SMAL datasets. It’s demonstrated that our method can achieve robust 3D pose transfer and be generalized to challenging meshes from unknown spaces on cross-dataset tasks. The code and dataset are made available. Code is available: https: //github. com/mikecheninoulu/CGT.

PDF Details

IS Journal 2022 Journal Article

Multiscale 3D-Shift Graph Convolution Network for Emotion Recognition From Human Actions

Henglin Shi
Wei Peng
Haoyu Chen
Xin Liu
Guoying Zhao

Emotion recognition from body gestures is challenging since similar emotions can be expressed by arbitrary spatial configurations of joints, which results in relying on modeling spatial-temporal patterns from a more global level. However, most recent powerful graph convolution networks (GCNs) separate the spatial and temporal modeling into isolated processes, where GCN models spatial interactions using partially fixed adjacent matrices and 1D convolution captures temporal dynamics, which is insufficient for emotion recognition. In this work, we propose the 3D-Shift GCN, which enables interactions of joints within a spatial-temporal volume for global feature extraction. Besides, we further develop a multiscale architecture, the MS-Shift GCN, to fuse features captured under different temporal ranges for modeling richer dynamics. After conducting evaluation on two regular action recognition benchmarks and two gesture based emotion recognition datasets, the results show that the proposed method outperforms several state-of-the-art methods.

Details DOI

IS Journal 2021 Journal Article

Adaptive Modality Distillation for Separable Multimodal Sentiment Analysis

Wei Peng
Xiaopeng Hong
Guoying Zhao

Multimodal sentiment analysis has increasingly attracted attention since with the arrival of complementary data streams, it has great potential to improve and go beyond unimodal sentiment analysis. In this article, we present an efficient separable multimodal learning method to deal with the tasks with modality missing issue. In this method, the multimodal tensor is utilized to guide the evolution of each separated modality representation. To save the computational expense, Tucker decomposition is introduced, which leads to a general extension of the low-rank tensor fusion method with more modality interactions. The method, in turn, enhances our modality distillation processing. Comprehensive experiments on three popular multimodal sentiment analysis datasets, CMU-MOSI, POM, and IEMOCAP, show a superior performance especially when only partial modalities are available.

Details DOI

IJCAI Conference 2021 Conference Paper

Dual-Cross Central Difference Network for Face Anti-Spoofing

Zitong Yu
Yunxiao Qin
Hengshuang Zhao
Xiaobai Li
Guoying Zhao

Face anti-spoofing (FAS) plays a vital role in securing face recognition systems. Recently, central difference convolution (CDC) has shown its excellent representation capacity for the FAS task via leveraging local gradient features. However, aggregating central difference clues from all neighbors/directions simultaneously makes the CDC redundant and sub-optimized in the training phase. In this paper, we propose two Cross Central Difference Convolutions (C-CDC), which exploit the difference of the center and surround sparse local features from the horizontal/vertical and diagonal directions, respectively. It is interesting to find that, with only five ninth parameters and less computational cost, C-CDC even outperforms the full directional CDC. Based on these two decoupled C-CDC, a powerful Dual-Cross Central Difference Network (DC-CDN) is established with Cross Feature Interaction Modules (CFIM) for mutual relation mining and local detailed representation enhancement. Furthermore, a novel Patch Exchange (PE) augmentation strategy for FAS is proposed via simply exchanging the face patches as well as their dense labels from random samples. Thus, the augmented samples contain richer live/spoof patterns and diverse domain distributions, which benefits the intrinsic and robust feature learning. Comprehensive experiments are performed on four benchmark datasets with three testing protocols to demonstrate our state-of-the-art performance.

PDF Details DOI

IJCAI Conference 2021 Conference Paper

Non-contact Pain Recognition from Video Sequences with Remote Physiological Measurements Prediction

Ruijing Yang
Ziyu Guan
Zitong Yu
Xiaoyi Feng
Jinye Peng
Guoying Zhao

Automatic pain recognition is paramount for medical diagnosis and treatment. The existing works fall into three categories: assessing facial appearance changes, exploiting physiological cues, or fusing them in a multi-modal manner. However, (1) appearance changes are easily affected by subjective factors which impedes objective pain recognition. Besides, the appearance-based approaches ignore long-range spatial-temporal dependencies that are important for modeling expressions over time; (2) the physiological cues are obtained by attaching sensors on human body, which is inconvenient and uncomfortable. In this paper, we present a novel multi-task learning framework which encodes both appearance changes and physiological cues in a non-contact manner for pain recognition. The framework is able to capture both local and long-range dependencies via the proposed attention mechanism for the learned appearance representations, which are further enriched by temporally attended physiological cues (remote photoplethysmography, rPPG) that are recovered from videos in the auxiliary task. This framework is dubbed rPPG-enriched Spatio-Temporal Attention Network (rSTAN) and allows us to establish the state-of-the-art performance of non-contact pain recognition on publicly available pain databases. It demonstrates that rPPG predictions can be used as an auxiliary task to facilitate non-contact automatic pain recognition.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Learning Graph Convolutional Network for Skeleton-Based Human Action Recognition by Neural Searching

Wei Peng
Xiaopeng Hong
Haoyu Chen
Guoying Zhao

Human action recognition from skeleton data, fuelled by the Graph Convolutional Network (GCN) with its powerful capability of modeling non-Euclidean data, has attracted lots of attention. However, many existing GCNs provide a pre-deﬁned graph structure and share it through the entire network, which can loss implicit joint correlations especially for the higherlevel features. Besides, the mainstream spectral GCN is approximated by one-order hop such that higher-order connections are not well involved. All of these require huge efforts to design a better GCN architecture. To address these problems, we turn to Neural Architecture Search (NAS) and propose the ﬁrst automatically designed GCN for this task. Speciﬁcally, we explore the spatial-temporal correlations between nodes and build a search space with multiple dynamic graph modules. Besides, we introduce multiple-hop modules and expect to break the limitation of representational capacity caused by one-order approximation. Moreover, a corresponding sampling- and memory-efﬁcient evolution strategy is proposed to search in this space. The resulted architecture proves the effectiveness of the higher-order approximation and the layer-wise dynamic graph modules. To evaluate the performance of the searched model, we conduct extensive experiments on two very large scale skeleton-based action recognition datasets. The results show that our model gets the stateof-the-art results in term of given metrics.

PDF Details

JBHI Journal 2017 Journal Article

HEp-2 Cell Classification via Combining Multiresolution Co-Occurrence Texture and Large Region Shape Information

Xianbiao Qi
Guoying Zhao
Chun-Guang Li
Jun Guo
Matti Pietikainen

Indirect immunofluorescence imaging of human epithelial type 2 (HEp-2) cell image is an effective evidence to diagnose autoimmune diseases. Recently, computer-aided diagnosis of autoimmune diseases by the HEp-2 cell classification has attracted great attention. However, the HEp-2 cell classification task is quite challenging due to large intraclass and small interclass variations. In this paper, we propose an effective approach for the automatic HEp-2 cell classification by combining multiresolution co-occurrence texture and large regional shape information. To be more specific, we propose to: 1) capture multiresolution co-occurrence texture information by a novel pairwise rotation-invariant co-occurrence of local Gabor binary pattern descriptor; 2) depict large regional shape information by using an improved Fisher vector model with RootSIFT features, which are sampled from large image patches in multiple scales; and 3) combine both features. We evaluate systematically the proposed approach on the IEEE International Conference on Pattern Recognition (ICPR) 2012, the IEEE International Conference on Image Processing (ICIP) 2013, and the ICPR 2014 contest datasets. The proposed method based on the combination of the introduced two features outperforms the winners of the ICPR 2012 contest using the same experimental protocol. Our method also greatly improves the winner of the ICIP 2013 contest under four different experimental setups. Using the leave-one-specimen-out evaluation strategy, our method achieves comparable performance with the winner of the ICPR 2014 contest that combined four features.

Details DOI