Author name cluster

Zhiheng Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers

2 author rows

EAAI Journal 2026 Journal Article

A dual-stream foreground-aware enhancement network with spiralscan-Mamba for vision-based occupancy prediction in autonomous driving

Rui Zheng
Nannan Liu
Yanyin Guo
Chuiyi Deng
Zhuoyi Zhao
Zhiheng Liu
Junwei Li

In autonomous driving perception, conventional three-dimensional object detection uses a closed set of categories and consequently fails to respond to previously unseen objects. By contrast, occupancy prediction estimates per-voxel occupancy and semantics, delivering finer geometric detail and approximate distance while generalizing beyond known classes to support downstream planning. Despite these advantages, prevailing occupancy architectures suffer from foreground–background imbalance and peripheral feature sparsity, which hinder accurate reasoning about the more critical foreground categories for autonomous driving and distant objects. To address these limitations, we propose a dual-stream framework that combines voxel-based modeling to strengthen foreground representation with a bird’s-eye-view stream for efficient global feature extraction, employs an adaptive foreground-global information fusion module to flexibly integrate complementary information across the two streams. Additionally, we introduce a Mamba-based spiral scanning mechanism that propagates structural priors from dense central areas to peripheral regions, effectively enriching boundary features. Experimental results demonstrate that our method achieves competitive state-of-the-art performance, with notably stronger foreground-category prediction and performance robust to low-resolution inputs. In particular, the proposed framework significantly enhances the representation and prediction of critical traffic participants such as vehicles and pedestrians, thereby improving the reliability of perception for real-world autonomous driving applications. The code will be publicly available at: https: //github. com/Bryan-ZhengRui/DFEOcc.

Details DOI

NeurIPS Conference 2025 Conference Paper

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Ruili Feng
Han Zhang
Zhilei Shu
Zhantao Yang
Longxiang Tang
Zhicai Wang
Andy Zheng
Jie Xiao

We present The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives. Trained on limited supervised data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains—deserts, grasslands, water bodies, and urban landscapes—in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting—an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

PDF Details

NeurIPS Conference 2025 Conference Paper

ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models

Zixun Fang
Kai Zhu
Zhiheng Liu
Yu Liu
Wei Zhai
Yang Cao
Zheng-Jun Zha

Panoramic video generation aims to synthesize 360-degree immersive videos, holding significant importance in the fields of VR, world models, and spatial intelligence. Existing works fail to synthesize high-quality panoramic videos due to the inherent modality gap between panoramic data and perspective data, which constitutes the majority of the training data for modern diffusion models. In this paper, we propose a novel framework utilizing pretrained perspective video models for generating panoramic videos. Specifically, we design a novel panorama representation named ViewPoint map, which possesses global spatial continuity and fine-grained visual details simultaneously. With our proposed Pano-Perspective attention mechanism, the model benefits from pretrained perspective priors and captures the panoramic spatial correlations of the ViewPoint map effectively. Extensive experiments demonstrate that our method can synthesize highly dynamic and spatially consistent panoramic videos, achieving state-of-the-art performance and surpassing previous methods.

PDF Details

NeurIPS Conference 2025 Conference Paper

WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception

Zhiheng Liu
Xueqing Deng
Shoufa Chen
Angtian Wang
Qiushan Guo
Mingfei Han
Zeyue Xue
Mengzhao Chen

Generative video modeling has made significant strides, yet ensuring structural and temporal consistency over long sequences remains a challenge. Current methods predominantly rely on RGB signals, leading to accumulated errors in object structure and motion over extended durations. To address these issues, we introduce WorldWeaver, a robust framework for long video generation that jointly models RGB frames and perceptual conditions within a unified long-horizon modeling scheme. Our training framework offers three key advantages. First, by jointly predicting perceptual conditions and color information from a unified representation, it significantly enhances temporal consistency and motion dynamics. Second, by leveraging depth cues, which we observe to be more resistant to drift than RGB, we construct a memory bank that preserves clearer contextual information, improving quality in long-horizon video generation. Third, we employ segmented noise scheduling for training prediction groups, which further mitigates drift and reduces computational cost. Extensive experiments on both diffusion and rectified flow-based models demonstrate the effectiveness of WorldWeaver in reducing temporal drift and improving the fidelity of generated videos.

PDF Details

ICML Conference 2024 Conference Paper

CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models

Jie Xiao 0002
Kai Zhu 0004
Han Zhang 0010
Zhiheng Liu
Yujun Shen
Zhantao Yang
Ruili Feng
Yu Liu 0063

Consistency Models (CMs) have showed a promise in creating high-quality images with few steps. However, the way to add new conditional controls to the pre-trained CMs has not been explored. In this paper, we explore the pivotal subject of leveraging the generative capacity and efficiency of consistency models to facilitate controllable visual content creation via ControlNet. First, it is observed that ControlNet trained for diffusion models (DMs) can be directly applied to CMs for high-level semantic controls but sacrifice image low-level details and realism. To tackle with this issue, we develop a CMs-tailored training strategy for ControlNet using the consistency training. It is substantiated that ControlNet can be successfully established through the consistency training technique. Besides, a unified adapter can be trained utilizing the consistency training, which enhances the adaptation of DM’s ControlNet. We quantitatively and qualitatively evaluate all strategies across various conditional controls, including sketch, hed, canny, depth, human pose, low-resolution image and masked image, with the pre-trained text-to-image latent consistency models.

Details

ICLR Conference 2024 Conference Paper

DreamClean: Restoring Clean Image Using Deep Diffusion Prior

Jie Xiao 0002
Ruili Feng
Han Zhang 0010
Zhiheng Liu
Zhantao Yang
Yurui Zhu
Xueyang Fu
Kai Zhu 0004

Image restoration poses a garners substantial interest due to the exponential surge in demands for recovering high-quality images from diverse mobile camera devices, adverse lighting conditions, suboptimal shooting environments, and frequent image compression for efficient transmission purposes. Yet this problem gathers significant challenges as people are blind to the type of restoration the images suffer, which, is usually the case in real-day scenarios and is most urgent to solve for this field. Current research, however, heavily relies on prior knowledge of the restoration type, either explicitly through rules or implicitly through the availability of degraded-clean image pairs to define the restoration process, and consumes considerable effort to collect image pairs of vast degradation types. This paper introduces DreamClean, a training-free method that needs no degradation prior knowledge but yields high-fidelity and generality towards various types of image degradation. DreamClean embeds the degraded image back to the latent of pre-trained diffusion models and re-sample it through a carefully designed diffusion process that mimics those generating clean images. Thanks to the rich image prior in diffusion models and our novel Variance Preservation Sampling (VPS) technique, DreamClean manages to handle various different degradation types at one time and reaches far more satisfied final quality than previous competitors. DreamClean relies on elegant theoretical supports to assure its convergence to clean image when VPS has appropriate parameters, and also enjoys superior experimental performance over various challenging tasks that could be overwhelming for previous methods when degradation prior is unavailable.

Details

AAAI Conference 2024 Conference Paper

Hypercorrelation Evolution for Video Class-Incremental Learning

Sen Liang
Kai Zhu
Wei Zhai
Zhiheng Liu
Yang Cao

Video class-incremental learning aims to recognize new actions while restricting the catastrophic forgetting of old ones, whose representative samples can only be saved in limited memory. Semantically variable subactions are susceptible to class confusion due to data imbalance. While existing methods address the problem by estimating and distilling the spatio-temporal knowledge, we further explores that the refinement of hierarchical correlations is crucial for the alignment of spatio-temporal features. To enhance the adaptability on evolved actions, we proposes a hierarchical aggregation strategy, in which hierarchical matching matrices are combined and jointly optimized to selectively store and retrieve relevant features from previous tasks. Meanwhile, a correlation refinement mechanism is presented to reinforce the bias on informative exemplars according to online hypercorrelation distribution. Experimental results demonstrate the effectiveness of the proposed method on three standard video class-incremental learning benchmarks, outperforming state-of-the-art methods. Code is available at: https://github.com/Lsen991031/HCE

PDF Details DOI

ICML Conference 2023 Conference Paper

Cones: Concept Neurons in Diffusion Models for Customized Generation

Zhiheng Liu
Ruili Feng
Kai Zhu 0004
Yifei Zhang
Kecheng Zheng
Yu Liu 0063
Deli Zhao
Jingren Zhou 0001

Human brains respond to semantic features of presented stimuli with different neurons. This raises the question of whether deep neural networks admit a similar behavior pattern. To investigate this phenomenon, this paper identifies a small cluster of neurons associated with a specific subject in a diffusion model. We call those neurons the concept neurons. They can be identified by statistics of network gradients to a stimulation connected with the given subject. The concept neurons demonstrate magnetic properties in interpreting and manipulating generation results. Shutting them can directly yield the related subject contextualized in different scenes. Concatenating multiple clusters of concept neurons can vividly generate all related concepts in a single image. Our method attains impressive performance for multi-subject customization, even four or more subjects. For large-scale applications, the concept neurons are environmentally friendly as we only need to store a sparse cluster of int index instead of dense float32 parameter values, reducing storage consumption by 90% compared with previous customized generation methods. Extensive qualitative and quantitative studies on diverse scenarios show the superiority of our method in interpreting and manipulating diffusion models.

Details

NeurIPS Conference 2023 Conference Paper

Customizable Image Synthesis with Multiple Subjects

Zhiheng Liu
Yifei Zhang
Yujun Shen
Kecheng Zheng
Kai Zhu
Ruili Feng
Yu Liu
Deli Zhao

Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, existing algorithms suffer from high training cost and low success rate along with increased number of subjects. Towards controllable image synthesis with multiple subjects as the constraints, this work studies how to efficiently represent a particular subject as well as how to appropriately compose different subjects. We find that the text embedding regarding the subject token already serves as a simple yet effective representation that supports arbitrary combinations without any model tuning. Through learning a residual on top of the base embedding, we manage to robustly shift the raw subject to the customized subject given various text conditions. We then propose to employ layout, a very abstract and easy-to-obtain prior, as the spatial guidance for subject arrangement. By rectifying the activations in the cross-attention map, the layout appoints and separates the location of different subjects in the image, significantly alleviating the interference across them. Using cross-attention map as the intermediary, we could strengthen the signal of target subjects and weaken the signal of irrelevant subjects within a certain region, significantly alleviating the interference across subjects. Both qualitative and quantitative experimental results demonstrate our superiority over state-of-the-art alternatives under a variety of settings for multi-subject customization.

PDF Details