Author name cluster

Hao Bai

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

EAAI Journal 2026 Journal Article

Theory-guided data-driven based on the learning curve for fracturing performance prediction

Yunjin Wang
Leyi Zheng
Gong Chen
Jianlong Zhang
Hao Bai
Hanxuan Song
Tingxue Jiang
Fujian Zhou

Accurate and robust prediction of fracturing performance is essential for optimizing fracturing strategies. Here, a fracturing learning curve is proposed based on the fracturing characteristics in Gimsar shale oil, and is used as a theoretical guide to build a theory-guided data-driven (TgDD) model to predict the fracturing performance. The fracturing learning curve is further decomposed into dimensionless trends and local fluctuations. Convolutional neural network (CNN) and gated recurrent unit (GRU) were combined to build a CNN-GRU to predict the dimensionless trend. Using adaptive boosting (AdaBoost) integrated random forest (RF) to build an AdaBoost-RF to predict the local fluctuations. The results show that dimensionless trend has time series characteristics. CNN-GRU can extract and select the features, and its prediction ability is 28. 1 % and 12. 9 % higher than that of CNN and GRU. AdaBoost-RF can dynamically adjust the weights, and its prediction ability is about 37% higher than that of the RF. TgDD is more sensitive to engineering parameters. Relative to the direct prediction, the prediction accuracy of the TgDD is improved by 47. 6 %. There are two main reasons for the higher prediction accuracy of TgDD. One is that the dimensionless trend belongs to the time series data, for which the established CNN-GRU model has an extremely strong prediction ability. The second is that the fluctuation amplitude of local fluctuations is reduced, which improves the data quality. The engineering parameters of the newly fractured wells were optimized using TgDD, and its estimated ultimate recovery was improved from 0. 4847 to 0. 4917.

Details DOI

ICLR Conference 2025 Conference Paper

Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents

Hao Bai
Yifei Zhou
Li Erran Li
Sergey Levine
Aviral Kumar

While a number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments (e.g., mobile device control). On-policy reinforcement learning (RL) should address these limitations, but collecting actual rollouts in an environment is often undesirable in truly open-ended agentic problems such as mobile device control or interacting with humans, where each unit of interaction is associated with a cost. In such scenarios, a method for policy learning that can utilize off-policy experience by learning a trained action-value function is much more effective. In this paper, we develop an approach, called Digi-Q, to train VLM-based action-value Q-functions which are then used to extract the agent policy. We study our approach in the mobile device control setting. Digi-Q trains the Q-function using offline temporal-difference (TD) learning, on top of frozen, intermediate-layer features of a VLM. Compared to fine-tuning the whole VLM, this approach saves us compute and enhances scalability. To make the VLM features amenable for representing the Q-function, we need to employ an initial phase of fine-tuning to amplify coverage over actionable information needed for value function. Once trained, we use this Q-function via a Best-of-N policy extraction operator that imitates the best action out of multiple candidate actions from the current policy as ranked by the value function, enabling policy improvement without environment interaction. Digi-Q outperforms several prior methods on user-scale device control tasks in Android-in-the-Wild, attaining 21.2% improvement over prior best-performing method. In some cases, our Digi-Q ap- proach already matches state-of-the-art RL methods that require interaction. The project is open-sourced at https://github.com/DigiRL-agent/digiq

Details

NeurIPS Conference 2025 Conference Paper

Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction

Junhong Shen
Hao Bai
Lunjun Zhang
Yifei Zhou
Amrith Setlur
Peter Tong
Diego Caples
Nan Jiang

Test-time scaling in agentic tasks often relies on generating long reasoning traces ("think" more) before acting, but this does not allow agents to acquire new information from the environment or adapt behavior over time. In this work, we propose scaling test-time interaction, an untapped dimension for test-time scaling that increases the agent's interaction horizon to enable rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we situate our study in the domain of web agents. We first show that even prompting-based interaction scaling can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI, a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their interaction lengths during rollout. Using a Gemma 3 12B model, TTI sets a new state-of-the-art among open-source agents trained on public data on WebVoyager and WebArena. Case studies further reveal that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-action compute, offering new avenues for training robust and adaptive agents.

PDF Details

NeurIPS Conference 2024 Conference Paper

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Hao Bai
Yifei Zhou
Mert Cemri
Jiayi Pan
Alane Suhr
Sergey Levine
Aviral Kumar

Pre-trained vision language models (VLMs), though powerful, typically lack training on decision-centric data, rendering them sub-optimal for decision-making tasks such as in-the-wild device control through Graphical User Interfaces (GUIs) when used off-the-shelf. While training with static demonstrations has shown some promise, we show that such methods fall short when controlling real GUIs due to their failure to deal with real world stochasticity and dynamism not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline and offline-to-online RL. We first build a scalable and parallelizable Android learning environment equipped with a VLM-based general-purpose evaluator and then identify the key design choices for simple and effective RL in this domain. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1. 5B VLM trained with RL achieves a 49. 5\% absolute improvement -- from 17. 7 to 67. 2\% success rate -- over supervised fine-tuning with static human demonstration data. It is worth noting that such improvement is achieved without any additional supervision or demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8. 3\% success rate) and the 17B CogAgent trained with AitW data (14. 4\%), but also our implementation of prior best autonomous RL approach based on filtered behavior cloning (57. 8\%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Yuexiang Zhai
Hao Bai
Zipeng Lin
Jiayi Pan
Shengbang Tong
Yifei Zhou
Alane Suhr
Saining Xie

Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.

PDF Details DOI

JMLR Journal 2024 Journal Article

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?

Yaodong Yu
Sam Buchanan
Druv Pai
Tianzhe Chu
Ziyang Wu
Shengbang Tong
Hao Bai
Yuexiang Zhai

In this paper, we contend that a natural objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a low-dimensional Gaussian mixture supported on incoherent subspaces. The goodness of such a representation can be evaluated by a principled measure, called sparse rate reduction, that simultaneously maximizes the intrinsic information gain and extrinsic sparsity of the learned representation. From this perspective, popular deep network architectures, including transformers, can be viewed as realizing iterative schemes to optimize this measure. Particularly, we derive a transformer block from alternating optimization on parts of this objective: the multi-head self-attention operator compresses the representation by implementing an approximate gradient descent step on the coding rate of the features, and the subsequent multi-layer perceptron sparsifies the features. This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable. We show, by way of a novel connection between denoising and compression, that the inverse to the aforementioned compressive encoding can be realized by the same class of CRATE architectures. Thus, the so-derived white-box architectures are universal to both encoders and decoders. Experiments show that these networks, despite their simplicity, indeed learn to compress and sparsify representations of large-scale real-world image and text datasets, and achieve strong performance across different settings: ViT, MAE, DINO, BERT, and GPT2. We believe the proposed computational framework demonstrates great potential in bridging the gap between theory and practice of deep learning, from a unified perspective of data compression. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2024. ( edit, beta )

PDF Details