Author name cluster

Qihao Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

10 papers

2 author rows

TMLR Journal 2026 Journal Article

ReVision: Refining Video Diffusion with Explicit 3D Motion Modeling

Qihao Liu
Ju He
Qihang Yu
Liang-Chieh Chen
Alan Yuille

In recent years, video generation has seen significant advancements. However, challenges still persist in generating complex motions and interactions. To address these challenges, we introduce ReVision, a plug-and-play framework that explicitly integrates parameterized 3D model knowledge into a pretrained conditional video generation model, significantly enhancing its ability to generate high-quality videos with complex motion and interactions. Specifically, ReVision consists of three stages. First, a video diffusion model is used to generate a coarse video. Next, we extract a set of 2D and 3D features from the coarse video to construct a 3D object-centric representation, which is then refined by our proposed parameterized motion prior model to produce an accurate 3D motion sequence. Finally, this refined motion sequence is fed back into the same video diffusion model as additional conditioning, enabling the generation of motion-consistent videos, even in scenarios involving complex actions and interactions. We validate the effectiveness of our approach on Stable Video Diffusion, where ReVision significantly improves motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even outperforms a state-of-the-art video generation model with over 13B parameters on complex video generation by a substantial margin. Our results suggest that, by incorporating 3D motion knowledge, even a relatively small video diffusion model can generate complex motions and interactions with greater realism and controllability, offering a promising solution for physically plausible video generation.

PDF Details

EAAI Journal 2025 Journal Article

A heterogeneous graph attention-enhanced deep reinforcement learning framework for flexible job shop scheduling problem with variable sublots

Zipeng Yang
Xinyu Li
Liang Gao
Qihao Liu

Variable lot-sizing is an effective approach to improve production efficiency by splitting an operation into several sublots, which has been widely applied in flexible manufacturing systems. However, the flexibility of lot-sizing will dramatically expand the solution space, leading to excessive computation time in converging to the relative optimum. To address this challenge, this paper introduces an end-to-end deep reinforcement learning framework based on heterogeneous graph attention mechanisms (HGADRL) for flexible job shop scheduling problem with variable sublots. Unlike traditional heuristic and rule-based methods, HGADRL dynamically learns the high-dimensional nature, providing a more generalizable solution in a very short time. In HGADRL, a modified heterogeneous disjunctive graph is designed to represent the dynamic scheduling status, including operation selection and sublot division. A dual-scale graph attention network combined with two interconnected attention modules is developed, enabling the precise capture of complex interdependencies between heterogeneous vertices. This approach can significantly enhance the agent's ability to self-learn and evolve optimal policies. By leveraging local and global features extracted through the graph attention network, an actor-critic network is employed for high-quality scheduling in different states. Experimental results demonstrate that the proposed method outperforms the 12 mixed priority dispatching rules, two meta-heuristic methods and two deep reinforcement learning methods in all 500 synthetic instances. Additionally, the proposed method outperforms all compared methods across 16 unseen scales of instances and four real-world instances, demonstrating its strong generalization capabilities.

Details DOI

IROS Conference 2025 Conference Paper

Automatic MILP Model Construction for Multi-Robot Task Allocation and Scheduling Based on Large Language Models

Mingming Peng
Zhendong Chen
Jie Yang
Jin Huang
Zhengqi Shi
Qihao Liu
Xinyu Li 0001
Liang Gao 0001

With the accelerated development of Industry 4. 0, intelligent manufacturing systems increasingly require efficient task allocation and scheduling in multi-robot systems. However, existing methods rely on domain expertise and face challenges in adapting to dynamic production constraints. Additionally, enterprises have high privacy requirements for production scheduling data, which prevents the use of cloud-based large language models (LLMs) for solution development. To address these challenges, there is an urgent need for an automated modeling solution that meets data privacy requirements. This study proposes a knowledge-augmented mixed integer linear programming (MILP) automated formulation framework, integrating local LLMs with domain-specific knowledge bases to generate executable code from natural language descriptions automatically. The framework employs a knowledge-guided DeepSeek-R1-Distill-Qwen-32B model to extract complex spatiotemporal constraints (82% average accuracy) and leverages a supervised fine-tuned Qwen2. 5-Coder-7B-Instruct model for efficient MILP code generation (90% average accuracy). Experimental results demonstrate that the framework successfully achieves automatic modeling in the aircraft skin manufacturing case while ensuring data privacy and computational efficiency. This research provides a low-barrier and highly reliable technical path for modeling in complex industrial scenarios.

Details

NeurIPS Conference 2025 Conference Paper

SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning

Wufei Ma
Yu-Cheng Chou
Qihao Liu
Xingrui Wang
Celso de Melo
Jianwen Xie
Alan Yuille

Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models. Recent studies explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual question-answering data. However, these methods typically perform spatial reasoning in an implicit manner and often fail on questions that are trivial to humans, even with long chain-of-thought reasoning. In this work, we introduce SpatialReasoner, a novel large vision-language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages--3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability to novel question types. Furthermore, by analyzing the explicit 3D representations in multi-step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMs. Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2. 0 by 9. 2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning.

PDF Details

NeurIPS Conference 2024 Conference Paper

Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization

Qihao Liu
Zhanpeng Zeng
Ju He
Qihang Yu
Xiaohui Shen
Liang-Chieh Chen

This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization. Diffusion models have gained prominence for their effectiveness in high-fidelity image generation. While conventional approaches rely on convolutional U-Net architectures, recent Transformer-based designs have demonstrated superior performance and scalability. However, Transformer architectures, which tokenize input data (via "patchification"), face a trade-off between visual fidelity and computational complexity due to the quadratic nature of self-attention operations concerning token length. While larger patch sizes enable attention computation efficiency, they struggle to capture fine-grained visual details, leading to image distortions. To address this challenge, we propose augmenting the **Di**ffusion model with the **M**ulti-**R**esolution network (DiMR), a framework that refines features across multiple resolutions, progressively enhancing detail from low to high resolution. Additionally, we introduce Time-Dependent Layer Normalization (TD-LN), a parameter-efficient approach that incorporates time-dependent parameters into layer normalization to inject time information and achieve superior performance. Our method's efficacy is demonstrated on the class-conditional ImageNet generation benchmark, where DiMR-XL variants surpass previous diffusion models, achieving FID scores of 1. 70 on ImageNet $256 \times 256$ and 2. 89 on ImageNet $512 \times 512$. Our best variant, DiMR-G, further establishes a state-of-the-art 1. 63 FID on ImageNet $256 \times 256$.

PDF Details DOI

ICLR Conference 2024 Conference Paper

Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search

Qihao Liu
Adam Kortylewski
Yutong Bai
Song Bai 0001
Alan L. Yuille

Text-guided diffusion models (TDMs) are widely applied but can fail unexpectedly. Common failures include: _(i)_ natural-looking text prompts generating images with the wrong content, or _(ii)_ different random samples of the latent variables that generate vastly different, and even unrelated, outputs despite being conditioned on the same text prompt. In this work, we aim to study and understand the failure modes of TDMs in more detail. To achieve this, we propose SAGE, the first adversarial search method on TDMs that systematically explores the discrete prompt space and the high-dimensional latent space, to automatically discover undesirable behaviors and failure cases in image generation. We use image classifiers as surrogate loss functions during searching, and employ human inspections to validate the identified failures. For the first time, our method enables efficient exploration of both the discrete and intricate human language space and the challenging latent space, overcoming the gradient vanishing problem. Then, we demonstrate the effectiveness of SAGE on five widely used generative models and reveal four typical failure modes that have not been systematically studied before: (1) We find a variety of natural text prompts that generate images failing to capture the semantics of input texts. We further discuss the underlying causes and potential solutions based on the results. (2) We find regions in the latent space that lead to distorted images independent of the text prompt, suggesting that parts of the latent space are not well-structured. (3) We also find latent samples that result in natural-looking images unrelated to the text prompt, implying a possible misalignment between the latent and prompt spaces. (4) By appending a single adversarial token embedding to any input prompts, we can generate a variety of specified target objects, with minimal impact on CLIP scores, demonstrating the fragility of language representations.

Details

ICLR Conference 2024 Conference Paper

Generating Images with 3D Annotations Using Diffusion Models

Wufei Ma
Qihao Liu
Jiahao Wang 0001
Angtian Wang
Xiaoding Yuan
Yi Zhang 0099
Zihao Xiao 0001
Guofeng Zhang 0020

Diffusion models have emerged as a powerful generative method, capable of producing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the generated images or to craft instances with specific poses and distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D geometry control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories~(e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically. This allows us to improve a wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out-of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100/200, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outperforms existing methods, e.g., 3.8 percentage points on ImageNet-100 using DeiT-B. Our code is available at <https://ccvl.jhu.edu/3D-DST/>

Details

NeurIPS Conference 2024 Conference Paper

ImageNet3D: Towards General-Purpose Object-Level 3D Understanding

Wufei Ma
Guofeng Zhang
Qihao Liu
Guanning Zeng
Adam Kortylewski
Yaoyao Liu
Alan Yuille

A vision model with general-purpose object-level 3D understanding should be capable of inferring both 2D ( e. g. , class name and bounding box) and 3D information ( e. g. , 3D location and 3D viewpoint) for arbitrary rigid objects in natural images. This is a challenging task, as it involves inferring 3D information from 2D signals and most importantly, generalizing to rigid objects from unseen categories. However, existing datasets with object-level 3D annotations are often limited by the number of categories or the quality of annotations. Models developed on these datasets become specialists for certain categories or domains, and fail to generalize. In this work, we present ImageNet3D, a large dataset for general-purpose object-level 3D understanding. ImageNet3D augments 200 categories from the ImageNet dataset with 2D bounding box, 3D pose, 3D location annotations, and image captions interleaved with 3D information. With the new annotations available in ImageNet3D, we could (i) analyze the object-level 3D awareness of visual foundation models, and (ii) study and develop general-purpose models that infer both 2D and 3D information for arbitrary rigid objects in natural images, and (iii) integrate unified 3D models with large language models for 3D-related reasoning. We consider two new tasks, probing of object-level 3D awareness and open vocabulary pose estimation, besides standard classification and pose estimation. Experimental results on ImageNet3D demonstrate the potential of our dataset in building vision models with stronger general-purpose object-level 3D understanding. Our dataset and project page are available here: https: //imagenet3d. github. io.

PDF Details DOI

EAAI Journal 2023 Journal Article

A multi-population co-evolutionary algorithm for green integrated process planning and scheduling considering logistics system

Qihao Liu
Cuiyu Wang
Xinyu Li
Liang Gao

Nowadays, green manufacturing is an increasingly significant theme in the worldwide industry fields. System integration can fully exploit the potential of subsystems to achieve satisfying green and low-carbon manufacturing. In particular, integrated process planning and scheduling (IPPS) can obtain better process routes and scheduling schemes to realize more efficient and less energy-consuming production by utilizing the complementary attributes of the process and scheduling subsystems. Additional consideration of the shop logistics system including task assignment of automated guided vehicles (AGVs) can improve shop productivity while ensuring the smooth running of the whole manufacturing system. Taking into account AGV transportation, this paper studies a green multi-objective IPPS problem considering logistics system (MO_IPPS_LS) to minimize the maximum completion time and energy consumption simultaneously. A multi-population co-evolutionary algorithm (MPCEA) is proposed with a novel integrated encoding method. The co-evolutionary framework can ensure the diversity of the populations through a backtracking mechanism and different evolutionary operators including modified critical-path based local searching. The proposed MPCEA is tested on the open benchmarks with different proportions of transport resources. The comparative experimental results show the effectiveness and superiority of MPCEA for solving MO_IPPS_LS problem.

Details DOI

IROS Conference 2021 Conference Paper

PNS: Population-Guided Novelty Search for Reinforcement Learning in Hard Exploration Environments

Qihao Liu
Yujia Wang
Xiaofeng Liu

Reinforcement Learning (RL) has made remarkable achievements, but it still suffers from inadequate exploration strategies, sparse reward signals, and deceptive reward functions. To alleviate these problems, a Population-guided Novelty Search (PNS) parallel learning method is proposed in this paper. In PNS, the population is divided into multiple sub-populations, each of which has one chief agent and several exploring agents. The chief agent evaluates the policies learned by exploring agents and shares the optimal policy with all sub-populations. The exploring agents learn their policies in collaboration with the guidance of the optimal policy and, simultaneously, upload their policies to the chief agent. To balance exploration and exploitation, the Novelty Search (NS) is employed in every chief agent to encourage policies with high novelty while maximizing per-episode performance. We apply PNS to the twin delayed deep deterministic (TD3) policy gradient algorithm. The effectiveness of PNS to promote exploration and improve performance in continuous control domains is demonstrated in the experimental section. Notably, PNS-TD3 achieves rewards that far exceed the SOTA methods in environments with sparse or delayed reward signals. We also demonstrate that PNS enables robotic agents to learn control policies directly from pixels for sparse-reward manipulation in both simulated and real-world settings.

Details