Author name cluster

Jiawei He

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

Wenyao Zhang
Hongsi Liu
Zekun Qi
Yunnan Wang
XinQiang Yu
Jiazhao Zhang
Runpei Dong
Jiawei He

Recent advances in vision-language-action (VLA) models have shown promise in integrating image generation with action prediction to improve generalization and reasoning in robot manipulation. However, existing methods are limited to challenging image-based forecasting, which suffers from redundant information and lacks comprehensive and critical world knowledge, including dynamic, spatial and semantic information. To address these limitations, we propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling, thereby establishing a perception-prediction-action loop for manipulation tasks. Specifically, DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning. This design aligns with how humans interact with the world by first forming abstract multimodal reasoning chains before acting. To mitigate interference among the dynamic, spatial and semantic information during training, we adopt a block-wise structured attention mechanism that masks their mutual attention, preventing information leakage and keeping each representation clean and disentangled. Moreover, to model the conditional distribution over future actions, we employ a diffusion-based transformer that disentangles action representations from shared latent features. Extensive experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76. 7 success rate on real robot tasks and 4. 44 average length on the CALVIN ABC-D benchmarks.

PDF Details

NeurIPS Conference 2025 Conference Paper

SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation

Zekun Qi
Wenyao Zhang
Yufei Ding
Runpei Dong
XinQiang Yu
Jingwen Li
Lingyun Xu
Baoyu Li

While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation—a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e. g. , the ''plug-in'' direction of a USB or the ''handle'' direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e. g. , zero-shot 48. 7\% successful rate on Open6DOR and zero-shot 74. 9\% successful rate on SIMPLER-Env.

PDF Details

ICLR Conference 2024 Conference Paper

AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval

Qi Yan
Raihan Seraj
Jiawei He
Lili Meng
Tristan Sylvain

Machine-based prediction of real-world events is garnering attention due to its potential for informed decision-making. Whereas traditional forecasting predominantly hinges on structured data like time-series, recent breakthroughs in language models enable predictions using unstructured text. In particular, (Zou et al., 2022) unveils AutoCast, a new benchmark that employs news articles for answering forecasting queries. Nevertheless, existing methods still trail behind human performance. The cornerstone of accurate forecasting, we argue, lies in identifying a concise, yet rich subset of news snippets from a vast corpus. With this motivation, we introduce AutoCast++, a zero-shot ranking-based context retrieval system, tailored to sift through expansive news document collections for event forecasting. Our approach first re-ranks articles based on zero-shot question-passage relevance, honing in on semantically pertinent news. Following this, the chosen articles are subjected to zero-shot summarization to attain succinct context. Leveraging a pre-trained language model, we conduct both the relevance evaluation and article summarization without needing domain-specific training. Notably, recent articles can sometimes be at odds with preceding ones due to new facts or unanticipated incidents, leading to fluctuating temporal dynamics. To tackle this, our re-ranking mechanism gives preference to more recent articles, and we further regularize the multi-passage representation learning to align with human forecaster responses made on different dates. Empirical results underscore marked improvements across multiple metrics, improving the performance for multiple-choice questions (MCQ) by 48% and true/false (TF) questions by up to 8%. Code is available at https://github.com/BorealisAI/Autocast-plus-plus.

Details

NeurIPS Conference 2024 Conference Paper

DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model

Yuqi Wang
Ke Cheng
Jiawei He
Qitai Wang
Hengchen Dai
Yuntao Chen
Fei Xia
Zhaoxiang Zhang

Driving world models have gained increasing attention due to their ability to model complex physical dynamics. However, their superb modeling capability is yet to be fully unleashed due to the limited video diversity in current driving datasets. We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. Our dataset features video clips with a complete set of driving maneuvers, diverse multi-agent interplay, and rich open-world driving knowledge, laying a stepping stone for future world model development. We further define an action instruction following (AIF) benchmark for world models and demonstrate the superiority of the proposed dataset for generating action-controlled future predictions.

PDF Details DOI

IJCAI Conference 2018 Conference Paper

ACV-tree: A New Method for Sentence Similarity Modeling

Yuquan Le
Zhi-Jie Wang
Zhe Quan
Jiawei He
Bin Yao

Sentence similarity modeling lies at the core of many natural language processing applications, and thus has received much attention. Owing to the success of word embeddings, recently, popular neural network methods have achieved sentence embedding, obtaining attractive performance. Nevertheless, most of them focused on learning semantic information and modeling it as a continuous vector, while the syntactic information of sentences has not been fully exploited. On the other hand, prior works have shown the benefits of structured trees that include syntactic information, while few methods in this branch utilized the advantages of word embeddings and another powerful technique? attention weight mechanism. This paper makes the first attempt to absorb their advantages by merging these techniques in a unified structure, dubbed as ACV-tree. Meanwhile, this paper develops a new tree kernel, known as ACVT kernel, that is tailored for sentence similarity measure based on the proposed structure. The experimental results, based on 19 widely-used datasets, demonstrate that our model is effective and competitive, compared against state-of-the-art models.

PDF Details

YNIMG Journal 2007 Journal Article

Automated template-based PET region of interest analyses in the aging brain

Felice T. Sun
Roberta A. Schriber
Joel M. Greenia
Jiawei He
Amy Gitcho
William J. Jagust

The definition of regions of interest for PET data analysis poses a number of complex problems. While studies have shown that regions drawn on a template can be appropriate for extracting data for normal healthy subjects, it is unclear how these results can be applied to different populations. In this study, we focused on the aging population and examined how different parameters in the template data-extraction process may affect the accuracy of the results. We first present an automated method for extracting PET counts using a region-of-interest approach within a template framework. Then, we discuss two studies in which we measure the effects of varying specific parameters in this process. In study 1 we examined three parameters that may influence this process: choice of template, region, and threshold. In study 2 we focused on the hippocampus. We considered 6 different templates, and examined how well the subject-specific hippocampal masks overlapped with each other and with the template hippocampal masks after normalization. While the data in the older cohort are more variable than the normal population, the results suggest that using an appropriate template and selecting the correct parameters for the template-based ROI method can provide template-extracted counts that are highly correlated to counts extracted using subject-specific ROIs.

Details DOI