Author name cluster

Liangyan Gui

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

TMLR Journal 2025 Journal Article

MoReact: Generating Reactive Motion from Textual Descriptions

Xiyan Xu
Sirui Xu
Yu-Xiong Wang
Liangyan Gui

Modeling and generating human reactions poses a significant challenge with broad applications for computer vision and human-computer interaction. Existing methods either treat multiple individuals as a single entity, directly generating interactions, or rely solely on one person's motion to generate the other's reaction, failing to integrate the rich semantic information that underpins human interactions. Yet, these methods often fall short in adaptive responsiveness, \ie, the ability to accurately respond to diverse and dynamic interaction scenarios. Recognizing this gap, our work introduces an approach tailored to address the limitations of existing models by focusing on text-driven human reaction generation. Our model specifically generates realistic motion sequences for individuals that responding to the other's actions based on a descriptive text of the interaction scenario. The goal is to produce motion sequences that not only complement the opponent's movements but also semantically fit the described interactions. To achieve this, we present MoReact, a diffusion-based method designed to disentangle the generation of global trajectories and local motions sequentially. This approach stems from the observation that generating global trajectories first is crucial for guiding local motion, ensuring better alignment with given action and text. Furthermore, we introduce a novel interaction loss to enhance the realism of generated close interactions. Our experiments, utilizing data adapted from a two-person motion dataset, demonstrate the efficacy of our approach for this novel task, which is capable of producing realistic, diverse, and controllable reactions that not only closely match the movements of the counterpart but also adhere to the textual guidance. Please find our webpage at https://xiyan-xu.github.io/MoReactWebPage.

PDF Details

ICLR Conference 2024 Conference Paper

SOHES: Self-supervised Open-world Hierarchical Entity Segmentation

Shengcao Cao
Jiuxiang Gu
Jason Kuen
Hao Tan 0002
Ruiyi Zhang 0002
Handong Zhao
Ani Nenkova
Liangyan Gui

Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive generalization capabilities on unseen images and concepts. Despite its promise, existing entity segmentation methods like Segment Anything Model (SAM) rely heavily on costly expert annotators. This work presents Self-supervised Open-world Hierarchical Entity Segmentation (SOHES), a novel approach that eliminates the need for human annotations. SOHES operates in three phases: self-exploration, self-instruction, and self-correction. Given a pre-trained self-supervised representation, we produce abundant high-quality pseudo-labels through visual feature clustering. Then, we train a segmentation model on the pseudo-labels, and rectify the noises in pseudo-labels via a teacher-student mutual-learning procedure. Beyond segmenting entities, SOHES also captures their constituent parts, providing a hierarchical understanding of visual entities. Using raw images as the sole training data, our method achieves unprecedented performance in self-supervised open-world segmentation, marking a significant milestone towards high-quality open-world entity segmentation in the absence of human-annotated masks. Project page: https://SOHES.github.io.

Details

IROS Conference 2023 Conference Paper

DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV Perception

Yunze Man
Liangyan Gui
Yu-Xiong Wang

Closing the domain gap between training and deployment and incorporating multiple sensor modalities are two challenging yet critical topics for self-driving. Existing work only focuses on single one of the above topics, overlooking the simultaneous domain and modality shift which pervasively exists in real-world scenarios. A model trained with multi-sensor data collected in Europe may need to run in Asia with a subset of input sensors available. In this work, we propose DualCross, a cross-modality cross-domain adaptation framework to facilitate the learning of a more robust monocular bird's-eye-view (BEV) perception model, which transfers the point cloud knowledge from a LiDAR sensor in one domain during the training phase to the camera-only testing scenario in a different domain. This work results in the first open analysis of cross-domain cross-sensor perception and adaptation for monocular 3D tasks in the wild. We benchmark our approach on large-scale datasets under a wide range of domain shifts and show state-of-the-art results against various baselines. Our project webpage is at https://yunzeman.github.io/DualCross.

Details

NeurIPS Conference 2023 Conference Paper

HASSOD: Hierarchical Adaptive Self-Supervised Object Detection

Shengcao Cao
Dhiraj Joshi
Liangyan Gui
Yu-Xiong Wang

The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self-supervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20. 2 to 22. 5 on LVIS, and from 17. 0 to 26. 0 on SA-1B. Project page: https: //HASSOD-NeurIPS23. github. io.

PDF Details

ICML Conference 2023 Conference Paper

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Shengcao Cao
Mengtian Li
James Hays
Deva Ramanan
Yu-Xiong Wang
Liangyan Gui

Resource-constrained perception systems such as edge computing and vision-for-robotics require vision models to be both accurate and lightweight in computation and memory usage. While knowledge distillation is a proven strategy to enhance the performance of lightweight classification models, its application to structured outputs like object detection and instance segmentation remains a complicated task, due to the variability in outputs and complex internal network modules involved in the distillation process. In this paper, we propose a simple yet surprisingly effective sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt. Our progressive strategy can be easily combined with existing detection distillation mechanisms to consistently maximize student performance in various settings. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36. 5% to 42. 0% AP and Mask R-CNN from 38. 2% to 42. 5% AP on the MS COCO benchmark. Code available at https: //github. com/Shengcao-Cao/MTPD.

Details

ICLR Conference 2023 Conference Paper

Stochastic Multi-Person 3D Motion Forecasting

Sirui Xu 0002
Yu-Xiong Wang
Liangyan Gui

This paper aims to deal with the ignored real-world complexities in prior work on human motion forecasting, emphasizing the social properties of multi-person motion, the diversity of motion and social interactions, and the complexity of articulated motion. To this end, we introduce a novel task of stochastic multi-person 3D motion forecasting. We propose a dual-level generative modeling framework that separately models independent individual motion at the local level and social interactions at the global level. Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through introducing learnable latent codes that represent intents of future motion and switching the codes' modes of operation at different levels. Our framework is general; we instantiate it with different generative models, including generative adversarial networks and diffusion models, and various multi-person forecasting models. Extensive experiments on CMU-Mocap, MuPoTS-3D, and SoMoF benchmarks show that our approach produces diverse and accurate multi-person predictions, significantly outperforming the state of the art.

Details

IROS Conference 2018 Conference Paper

Teaching Robots to Predict Human Motion

Liangyan Gui
Kevin Zhang 0002
Yu-Xiong Wang
Xiaodan Liang
José M. F. Moura
Manuela Veloso

Teaching a robot to predict and mimic how a human moves or acts in the near future by observing a series of historical human movements is a crucial first step in human-robot interaction and collaboration. In this paper, we instrument a robot with such a prediction ability by leveraging recent deep learning and computer vision techniques. First, our system takes images from the robot camera as input to produce the corresponding human skeleton based on real-time human pose estimation obtained with the OpenPose library. Then, conditioning on this historical sequence, the robot forecasts plausible motion through a motion predictor, generating a corresponding demonstration. Because of a lack of high-level fidelity validation, existing forecasting algorithms suffer from error accumulation and inaccurate prediction. Inspired by generative adversarial networks (GANs), we introduce a global discriminator that examines whether the predicted sequence is smooth and realistic. Our resulting motion GAN model achieves superior prediction performance to state-of-the-art approaches when evaluated on the standard H3. 6M dataset. Based on this motion GAN model, the robot demonstrates its ability to replay the predicted motion in a human-like manner when interacting with a person.

Details

YNIMG Journal 2017 Journal Article

Geodesic distance on a Grassmannian for monitoring the progression of Alzheimer's disease

Liangyan Gui
Xiaoying Tang
José M.F. Moura

We propose a geodesic distance on a Grassmannian manifold that can be used to quantify the shape progression patterns of the bilateral hippocampi, amygdalas, and lateral ventricles in healthy control (HC), mild cognitive impairment (MCI), and Alzheimer's disease (AD). Longitudinal magnetic resonance imaging (MRI) scans of 754 subjects (3092 scans in total) were used in this study. Longitudinally, the geodesic distance was found to be proportional to the elapsed time separating the two scans in question. Cross-sectionally, utilizing a linear mixed-effects statistical model, we found that each structure’s annualized rate of change in the geodesic distance followed the order of AD>MCI>HC, with statistical significance being reached in every case. In addition, for each of the six structures of interest, within the same time interval (e. g. , from baseline to the 6th month), we observed significant correlations between the geodesic distance and the cognitive deterioration as quantified by the ADAS-cog increase and the MMSE decrease. Furthermore, as the disease progresses over time, this linkage between the inter-shape geodesic distance and the cognitive decline becomes considerably stronger and more significant.

Details DOI