Author name cluster

Dongbin Zhao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

37 papers

2 author rows

IROS Conference 2025 Conference Paper

Advancing Object-Goal Navigation through LLM-enhanced Object Affinities Transfer

Mengying Lin
Shugao Liu
Dingxi Zhang
Yaran Chen
Zhaoran Wang 0001
Haoran Li 0010
Dongbin Zhao

Object-goal navigation requires mobile robots to efficiently locate targets with visual and spatial information, yet existing methods struggle with generalization in unseen environments. Heuristic approaches with naive metrics fail in complex layouts, while graph-based and learning-based methods suffer from environmental biases and limited generalization. Although Large Language Models (LLMs) as planners or agents offer a rich knowledge base, they are cost-inefficient and lack targeted historical experience. To address these challenges, we propose the LLM-enhanced Object Affinities Transfer (LOAT) framework, integrating LLM-derived semantics with learning-based approaches to leverage experiential object affinities for better generalization in unseen settings. LOAT employs a dual-module strategy: one module accesses LLMs’ vast knowledge, and the other applies learned object semantic relationships, dynamically fusing these sources based on context. Evaluations in AI2-THOR and Habitat simulators show significant improvements in navigation success and efficiency, and real-world deployment demonstrates the zero-shot ability of LOAT to enhance object-goal navigation systems.

AAMAS Conference 2025 Conference Paper

Consistency Policy with Categorical Critic for Autonomous Driving

Xing Fang
Qichao Zhang
Haoran Li
Dongbin Zhao

In the domain of autonomous driving, employing reinforcement learning (RL) for decision-making must effectively capture the range of feasible actions and accurately predict their consequences. The classical actor-critic framework in RL achieves this through an actor that selects actions and a critic that evaluates their values. However, traditional Gaussian-distributed actors are limited to learning unimodal distributions, which limits their ability to fully represent the diversity of executable actions that can be learned from past interactions. Moreover, the mean squared error (MSE) loss often employed by the critic is prone to significant estimation biases due to the non-stationary nature of RL training, leading to inaccurate assessments of future outcomes. In this paper, we introduce Consistency Policy with Categorical Critic (CPCC), a novel approach that leverages recent advancements in diffusion models, particularly consistency models, to serve as the actor, enabling the representation of multimodal action distributions. Additionally, we utilize classification loss (cross-entropy loss) for training the categorical critic, which mitigates overfitting to noisy targets and yields more precise approximations of Q-values. Experimental results obtained from the simulated driving environment MetaDrive substantiate the effectiveness of our proposed method. Code is available at https: //github. com/weiaiF/cpcc.

ICML Conference 2025 Conference Paper

Constrained Exploitability Descent: An Offline Reinforcement Learning Method for Finding Mixed-Strategy Nash Equilibrium

Runyu Lu
Yuanheng Zhu
Dongbin Zhao

This paper proposes Constrained Exploitability Descent (CED), a model-free offline reinforcement learning (RL) algorithm for solving adversarial Markov games (MGs). CED combines the game-theoretical approach of Exploitability Descent (ED) with policy constraint methods from offline RL. While policy constraints can perturb the optimal pure-strategy solutions in single-agent scenarios, we find the side effect less detrimental in adversarial games, where the optimal policy can be a mixed-strategy Nash equilibrium. We theoretically prove that, under the uniform coverage assumption on the dataset, CED converges to a stationary point in deterministic two-player zero-sum Markov games. We further prove that the min-player policy at the stationary point follows the property of mixed-strategy Nash equilibrium in MGs. Compared to the model-based ED method that optimizes the max-player policy, our CED method no longer relies on a generalized gradient. Experiments in matrix games, a tree-form game, and an infinite-horizon soccer game verify that CED can find an equilibrium policy for the min-player as long as the offline dataset guarantees uniform coverage. Besides, CED achieves a significantly lower NashConv compared to an existing pessimism-based method and can gradually improve the behavior policy even under non-uniform data coverages. When combined with neural networks, CED also outperforms behavior cloning and offline self-play in a large-scale two-team robotic combat game.

ICML Conference 2025 Conference Paper

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy

Kaixuan Xu
Jiajun Chai
Sicheng Li
Yuqian Fu
Yuanheng Zhu
Dongbin Zhao

Diplomacy is a complex multiplayer game that re- quires both cooperation and competition, posing significant challenges for AI systems. Traditional methods rely on equilibrium search to generate extensive game data for training, which demands substantial computational resources. Large Lan- guage Models (LLMs) offer a promising alterna- tive, leveraging pre-trained knowledge to achieve strong performance with relatively small-scale fine-tuning. However, applying LLMs to Diplo- macy remains challenging due to the exponential growth of possible action combinations and the intricate strategic interactions among players. To address this challenge, we propose DipLLM, a fine-tuned LLM-based agent that learns equilib- rium policies for Diplomacy. DipLLM employs an autoregressive factorization framework to sim- plify the complex task of multi-unit action assign- ment into a sequence of unit-level decisions. By defining an equilibrium policy within this frame- work as the learning objective, we fine-tune the model using only 1. 5% of the data required by the state-of-the-art Cicero model, surpassing its per- formance. Our results demonstrate the potential of fine-tuned LLMs for tackling complex strategic decision-making in multiplayer games.

ICLR Conference 2025 Conference Paper

Divergence-Regularized Discounted Aggregation: Equilibrium Finding in Multiplayer Partially Observable Stochastic Games

Runyu Lu
Yuanheng Zhu
Dongbin Zhao

This paper presents Divergence-Regularized Discounted Aggregation (DRDA), a multi-round learning system for solving partially observable stochastic games (POSGs). DRDA is based on action values and applicable to multiplayer POSGs, which can unify normal-form games (NFGs), extensive-form games (EFGs) with perfect recall, and Markov games (MGs). In each single round, DRDA can be viewed as a discounted variant of Follow the Regularized Leader (FTRL) under a general value function for POSGs. While previous studies on discounted FTRL have demonstrated its last-iterate convergence towards quantal response equilibrium (QRE) in NFGs, this paper extends the theoretical results to POSGs under divergence regularization and generalizes the QRE concept of Nash distribution. The linear last-iterate convergence of single-round DRDA to its rest point is proved under the assumption on the hypomonotonicity of the game. When the rest point is unique, it induces the unique Nash distribution defined in the POSG, which has a bounded deviation from Nash equilibrium (NE). Under multiple learning rounds, DRDA keeps replacing the base policy for divergence regularization with the policy at the rest point in the previous round. It is further proved that the limit point of multi-round DRDA must be an exact NE (rather than a QRE). In experiments, discrete-time DRDA can converge to NE at a near-exponential rate in (multiplayer) NFGs and outperform the existing baselines for EFGs, MGs, and typical POSGs.

ICLR Conference 2025 Conference Paper

Empowering LLM Agents with Zero-Shot Optimal Decision-Making through Q-learning

Jiajun Chai
Sicheng Li
Yuqian Fu
Dongbin Zhao
Yuanheng Zhu

Large language models (LLMs) are trained on extensive text data to gain general comprehension capability. Current LLM agents leverage this ability to make zero- or few-shot decisions without reinforcement learning (RL) but fail in making optimal decisions, as LLMs inherently perform next-token prediction rather than maximizing rewards. In contrast, agents trained via RL could make optimal decisions but require extensive environmental interaction. In this work, we develop an algorithm that combines the zero-shot capabilities of LLMs with the optimal decision-making of RL, referred to as the Model-based LLM Agent with Q-Learning (MLAQ). MLAQ employs Q-learning to derive optimal policies from transitions within memory. However, unlike RL agents that collect data from environmental interactions, MLAQ constructs an imagination space fully based on LLM to perform imaginary interactions for deriving zero-shot policies. Our proposed UCB variant generates high-quality imaginary data through interactions with the LLM-based world model, balancing exploration and exploitation while ensuring a sub-linear regret bound. Additionally, MLAQ incorporates a mixed-examination mechanism to filter out incorrect data. We evaluate MLAQ in benchmarks that present significant challenges for existing LLM agents. Results show that MLAQ achieves a optimal rate of over 90\% in tasks where other methods struggle to succeed. Additional experiments are conducted to reach the conclusion that introducing model-based RL into LLM agents shows significant potential to improve optimal decision-making ability. Our interactive website is available at http://mlaq.site.

NeurIPS Conference 2025 Conference Paper

Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games

Runyu Lu
Peng Zhang
Ruochuan Shi
Yuanheng Zhu
Dongbin Zhao
Yang Liu
Dong Wang
Cesare Alippi

Equilibrium learning in adversarial games is an important topic widely examined in the fields of game theory and reinforcement learning (RL). Pursuit-evasion game (PEG), as an important class of real-world games from the fields of robotics and security, requires exponential time to be accurately solved. When the underlying graph structure varies, even the state-of-the-art RL methods require recomputation or at least fine-tuning, which can be time-consuming and impair real-time applicability. This paper proposes an Equilibrium Policy Generalization (EPG) framework to effectively learn a generalized policy with robust cross-graph zero-shot performance. In the context of PEGs, our framework is generally applicable to both pursuer and evader sides in both no-exit and multi-exit scenarios. These two generalizability properties, to our knowledge, are the first to appear in this domain. The core idea of the EPG framework is to train an RL policy across different graph structures against the equilibrium policy for each single graph. To construct an equilibrium oracle for single-graph policies, we present a dynamic programming (DP) algorithm that provably generates pure-strategy Nash equilibrium with near-optimal time complexity. To guarantee scalability with respect to pursuer number, we further extend DP and RL by designing a grouping mechanism and a sequence model for joint policy decomposition, respectively. Experimental results show that, using equilibrium guidance and a distance feature proposed for cross-graph PEG training, the EPG framework guarantees desirable zero-shot performance in various unseen real-world graphs. Besides, when trained under an equilibrium heuristic proposed for the graphs with exits, our generalized pursuer policy can even match the performance of the fine-tuned policies from the state-of-the-art PEG methods.

AAAI Conference 2025 Conference Paper

In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning

Songjun Tu
Jingbo Sun
Qichao Zhang
Yaocheng Zhang
Jia Liu
Ke Chen
Dongbin Zhao

Offline preference-based reinforcement learning (PbRL) typically operates in two phases: first, use human preferences to learn a reward model and annotate rewards for a reward-free offline dataset; second, learn a policy by optimizing the learned reward via offline RL. However, accurately modeling step-wise rewards from trajectory-level preference feedback presents inherent challenges. The reward bias introduced, particularly the overestimation of predicted rewards, leads to optimistic trajectory stitching, which undermines the pessimism mechanism critical to the offline RL phase. To address this challenge, we propose In-Dataset Trajectory Return Regularization (DTR) for offline PbRL, which leverages conditional sequence modeling to mitigate the risk of learning inaccurate trajectory stitching under reward bias. Specifically, DTR employs Decision Transformer and TD-Learning to strike a balance between maintaining fidelity to the behavior policy with high in-dataset trajectory returns and selecting optimal actions based on high reward labels. Additionally, we introduce an ensemble normalization technique that effectively integrates multiple reward models, balancing the trade-off between reward differentiation and accuracy. Empirical evaluations on various benchmarks demonstrate the superiority of DTR over other state-of-the-art baselines.

PDF Details DOI

ICLR Conference 2025 Conference Paper

INS: Interaction-aware Synthesis to Enhance Offline Multi-agent Reinforcement Learning

Yuqian Fu
Yuanheng Zhu
Jian Zhao
Jiajun Chai
Dongbin Zhao

Data scarcity in offline multi-agent reinforcement learning (MARL) is a key challenge for real-world applications. Recent advances in offline single-agent reinforcement learning (RL) demonstrate the potential of data synthesis to mitigate this issue. However, in multi-agent systems, interactions between agents introduce additional challenges. These interactions complicate the synthesis of multi-agent datasets, leading to data distortion when inter-agent interactions are neglected. Furthermore, the quality of the synthetic dataset is often constrained by the original dataset. To address these challenges, we propose **INteraction-aware Synthesis (INS)**, which synthesizes high-quality multi-agent datasets using diffusion models. Recognizing the sparsity of inter-agent interactions, INS employs a sparse attention mechanism to capture these interactions, ensuring that the synthetic dataset reflects the underlying agent dynamics. To overcome the limitation of diffusion models requiring continuous variables, INS implements a bit action module, enabling compatibility with both discrete and continuous action spaces. Additionally, we incorporate a select mechanism to prioritize transitions with higher estimated values, further enhancing the dataset quality. Experimental results across multiple datasets in MPE and SMAC environments demonstrate that INS consistently outperforms existing methods, resulting in improved downstream policy performance and superior dataset metrics. Notably, INS can synthesize high-quality data using only 10% of the original dataset, highlighting its efficiency in data-limited scenarios.

NeurIPS Conference 2025 Conference Paper

Learning and Planning Multi-Agent Tasks via an MoE-based World Model

Zijie Zhao
Zhongyue Zhao
Kaixuan Xu
Yuqian Fu
Jiajun Chai
Yuanheng Zhu
Dongbin Zhao

Multi-task multi-agent reinforcement learning (MT-MARL) aims to develop a single model capable of solving a diverse set of tasks. However, existing methods often fall short due to the substantial variation in optimal policies across tasks, making it challenging for a single policy model to generalize effectively. In contrast, we find that many tasks exhibit bounded similarity in their underlying dynamics—highly similar within certain groups (e. g. , door-open/close) diverge significantly between unrelated tasks (e. g. , door-open & object-catch). To leverage this property, we reconsider the role of modularity in multi-task learning, and propose M3W, a novel approach that applies mixture-of-experts (MoE) to world model instead of policy, enabling both learning and planning. For learning, it uses a SoftMoE-based dynamics model alongside a SparseMoE-based predictor to facilitate knowledge reuse across similar tasks while avoiding gradient conflicts across dissimilar tasks. For planning, it evaluates and optimizes actions using the predicted rollouts from the world model, without relying directly on a explicit policy model, thereby overcoming the limitations of policy-centric methods. As the first MoE-based multi-task world model, M3W demonstrates superior performance, sample efficiency, and multi-task adaptability, as validated on Bi-DexHands with 14 tasks and MA-Mujoco with 24 tasks. The demos and anonymous code are available at \url{https: //github. com/zhaozijie2022/m3w-marl}.

NeurIPS Conference 2025 Conference Paper

Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

Songjun Tu
Jiahao Lin
Qichao Zhang
Xiangyu Tian
Linjing Li
Xiangyuan Lan
Dongbin Zhao

Large reasoning models (LRMs) are proficient at generating explicit, step-by-step reasoning sequences before producing final answers. However, such detailed reasoning can introduce substantial computational overhead and latency, particularly for simple problems. To address this over-thinking problem, we explore how to equip LRMs with adaptive thinking capabilities—enabling them to dynamically decide whether or not to engage in explicit reasoning based on problem complexity. Building on R1-style distilled models, we observe that inserting a simple ellipsis (". .. ") into the prompt can stochastically trigger either a thinking or no-thinking mode, revealing a latent controllability in the reasoning behavior. Leveraging this property, we propose AutoThink, a multi-stage reinforcement learning (RL) framework that progressively optimizes reasoning policies via stage-wise reward shaping. AutoThink learns to invoke explicit reasoning only when necessary, while defaulting to succinct responses for simpler tasks. Experiments on five mainstream mathematical benchmarks demonstrate that AutoThink achieves favorable accuracy–efficiency trade-offs compared to recent prompting and RL-based pruning methods. It can be seamlessly integrated into any R1-style model, including both distilled and further fine-tuned variants. Notably, AutoThink improves relative accuracy by 6. 4\% while reducing token usage by 52\% on DeepSeek-R1-Distill-Qwen-1. 5B, establishing a scalable and adaptive reasoning paradigm for LRMs. Project Page: https: //github. com/ScienceOne-AI/AutoThink.

AAMAS Conference 2025 Conference Paper

Offline Goal-Conditioned Reinforcement Learning with Elastic-Subgoal Diffused Policy Learning

Yaocheng Zhang
Yuanheng Zhu
Yuqian Fu
Songjun Tu
Dongbin Zhao

Goal-conditioned reinforcement learning (GCRL) aims to learn a policy that generalizes across different goal conditions. Compared to non-hierarchical methods, hierarchical GCRL based on subgoals can alleviate the problem of inaccurately estimating the value function for faraway goals in offline learning scenarios, thereby leading to more effective policy learning. Due to the state complexity of the decision-making process, at different states, we require subgoals from varying future time steps to minimize policy errors caused by noisy value functions, rather than using a fixed future time step for selecting subgoals. Therefore, we propose a hierarchical reinforcement learning algorithm with an elastic subgoal steps, called ESD (Elastic Subgoal Diffused Policy Learning). Our method defines a novel high-level policy in which all reachable states surrounding the current state are considered as potential subgoals, and the optimal subgoal is selected among them. Moreover, we use diffusion models to represent the hierarchical policies, enhancing their ability to capture the multimodal data distribution introduced by the elastic subgoal steps and offline data. We evaluate the performance of ESD on multiple goal-conditioned benchmarks, and it demonstrates superior performance compared to previous baselines. Our method effectively reduces the impact of inaccurate value function estimates on policy accuracy, especially in complex tasks and high-dimensional image observations. Code is available at https: //github. com/zhyaoch/ESD. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), Y. Vorobeychik, S. Das, A. Nowé (eds.), May 19 – 23, 2025, Detroit, Michigan, USA. © 2025 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org).

AAMAS Conference 2025 Conference Paper

Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

Songjun Tu
Jingbo Sun
Qichao Zhang
Xiangyuan Lan
Dongbin Zhao

Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences. However, real-time human feedback is hard to obtain in online tasks. Most work suppose there is a "scripted teacher" that utilizes privileged predefined reward to provide preference feedback. In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL. RL-SaLLM-F leverages the reflective and discriminative capabilities of LLM to generate self-augmented trajectories and provide preference labels for reward learning. First, we identify a failure issue in LLM-based preference discrimination, specifically "query ambiguity", in online PbRL. Then LLM is employed to provide preference labels and generate self-augmented imagined trajectories that better achieve the task goal, thereby enhancing the quality and efficiency of feedback. Additionally, a double-check mechanism is introduced to mitigate randomness in the preference labels, improving the reliability of LLM feedback. The experiment across multiple tasks in the MetaWorld benchmark demonstrates the specific contributions of each proposed module in RL-SaLLM-F, and shows that self-augmented LLM feedback can effectively replace the impractical "scripted teacher" feedback. In summary, RL-SaLLM-F introduces a new direction of feedback acquisition in online PbRL that does not rely on any online privileged information, offering an efficient and lightweight solution with LLM-driven feedback. 1 1Corresponding author: Qichao Zhang (zhangqichao2014@ia. ac. cn) and Xiangyuan Lan (lanxy@pcl. ac. cn). Code Page: https: //github. com/TU2021/RL-SaLLM-F This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), Y. Vorobeychik, S. Das, A. Nowé (eds.), May 19 – 23, 2025, Detroit, Michigan, USA. © 2025 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org).

AAMAS Conference 2025 Conference Paper

Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning

Jingbo Sun
Songjun Tu
Qichao Zhang
Ke Chen
Dongbin Zhao

Generalizing policies to unseen scenarios remains a critical challenge in visual reinforcement learning, where agents often overfit to the specific visual observations of the training environment. In unseen environments, distracting pixels may lead agents to extract representations containing task-irrelevant information. As a result, agents may deviate from the optimal behaviors learned during training, thereby hindering visual generalization. To address this issue, we propose the Salience-Invariant Consistent Policy Learning (SCPL) algorithm, an efficient framework for zero-shot generalization. Our approach introduces a novel value consistency module alongside a dynamics module to effectively capture taskrelevant representations. The value consistency module, guided by saliency, ensures the agent focuses on task-relevant pixels in both original and perturbed observations, while the dynamics module uses augmented data to help the encoder capture dynamicand reward-relevant representations. Additionally, our theoretical analysis highlights the importance of policy consistency for generalization. To strengthen this, we introduce a policy consistency module with a KL divergence constraint to maintain consistent policies across original and perturbed observations. Extensive experiments on the DMC-GB, Robotic Manipulation, and CARLA benchmarks demonstrate that SCPL significantly outperforms state-of-the-art methods in terms of generalization. Notably, SCPL achieves average performance improvements of 14%, 39%, and 69% in the challenging DMC video hard setting, the Robotic hard setting, and the CARLA benchmark, respectively. Project Page: https: //sites. google. com/view/scpl-rl. Corresponding author: Qichao Zhang, Dongbin Zhao. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), Y. Vorobeychik, S. Das, A. Nowé (eds.), May 19 – 23, 2025, Detroit, Michigan, USA. © 2025 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org).

TMLR Journal 2025 Journal Article

SELU: Self-Learning Embodied Multimodal Large Language Models in Unknown Environments

Boyu Li
Haobin Jiang
Ziluo Ding
Xinrun Xu
Haoran Li
Dongbin Zhao
Zongqing Lu

Recently, multimodal large language models (MLLMs) have demonstrated strong visual understanding and decision-making capabilities, enabling the exploration of autonomously improving MLLMs in unknown environments. However, external feedback like human or environmental feedback is not always available. To address this challenge, existing methods primarily focus on enhancing the decision-making capabilities of MLLMs through voting and scoring mechanisms, while little effort has been paid to improving the environmental comprehension of MLLMs in unknown environments. To fully unleash the self-learning potential of MLLMs, we propose a novel actor-critic self-learning paradigm, dubbed SELU, inspired by the actor-critic paradigm in reinforcement learning. The critic employs self-asking and hindsight relabeling to extract knowledge from interaction trajectories collected by the actor, thereby augmenting its environmental comprehension. Simultaneously, the actor is improved by the self-feedback provided by the critic, enhancing its decision-making. We evaluate our method in the AI2-THOR and VirtualHome environments, and SELU achieves critic improvements of approximately 28% and 30%, and actor improvements of about 20% and 24% via self-learning.

ICRA Conference 2025 Conference Paper

UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertainty

Pengxuan Yang
Yupeng Zheng
Qichao Zhang
Kefei Zhu
Zebin Xing
Qiao Lin
Yun-Fu Liu
Zhiguo Su

End-to-end autonomous driving aims to produce planning trajectories from raw sensors directly. Currently, most approaches integrate perception, prediction, and planning modules into a fully differentiable network, promising great scalability. However, these methods typically rely on deterministic modeling of online maps in the perception module for guiding or constraining vehicle planning, which may incorporate erroneous perception information and further compromise planning safety. To address this issue, we delve into the importance of online map uncertainty for enhancing autonomous driving safety and propose a novel paradigm named UncAD. Specifically, UncAD first estimates the uncertainty of the online map in the perception module. It then leverages the uncertainty to guide motion prediction and planning modules to produce multi-modal trajectories. Finally, to achieve safer autonomous driving, UncAD proposes an uncertainty-collision-aware planning selection strategy according to the online map uncertainty to evaluate and select the best trajectory. In this study, we incorporate UncAD into various state-of-the-art (SOTA) end-to-end methods. Experiments on the nuScenes dataset show that integrating UncAD, with only a 1. 9% increase in parameters, can reduce collision rates by up to 26% and drivable area conflict rate by up to 42%. Codes, pre-trained models, and demo videos can be accessed at https://github.com/pengxuanyang/UncAD.

ICLR Conference 2025 Conference Paper

Unsupervised Zero-Shot Reinforcement Learning via Dual-Value Forward-Backward Representation

Jingbo Sun
Songjun Tu
Qichao Zhang
Haoran Li 0010
Xin Liu 0039
Yaran Chen
Ke Chen
Dongbin Zhao

Online unsupervised reinforcement learning (URL) can discover diverse skills via reward-free pre-training and exhibits impressive downstream task adaptation abilities through further fine-tuning. However, online URL methods face challenges in achieving zero-shot generalization, i.e., directly applying pre-trained policies to downstream tasks without additional planning or learning. In this paper, we propose a novel Dual-Value Forward-Backward representation (DVFB) framework with a contrastive entropy intrinsic reward to achieve both zero-shot generalization and fine-tuning adaptation in online URL. On the one hand, we demonstrate that poor exploration in forward-backward representations can lead to limited data diversity in online URL, impairing successor measures, and ultimately constraining generalization ability. To address this issue, the DVFB framework learns successor measures through a skill value function while promoting data diversity through an exploration value function, thus enabling zero-shot generalization. On the other hand, and somewhat surprisingly, by employing a straightforward dual-value fine-tuning scheme combined with a reward mapping technique, the pre-trained policy further enhances its performance through fine-tuning on downstream tasks, building on its zero-shot performance. Through extensive multi-task generalization experiments, DVFB demonstrates both superior zero-shot generalization (outperforming on all 12 tasks) and fine-tuning adaptation (leading on 10 out of 12 tasks) abilities, surpassing state-of-the-art URL methods.

NeurIPS Conference 2025 Conference Paper

Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations

Xin Liu
Haoran Li
Dongbin Zhao

Humans can efficiently extract knowledge and learn skills from the videos within only a few trials and errors. However, it poses a big challenge to replicate this learning process for autonomous agents, due to the complexity of visual input, the absence of action or reward signals, and the limitations of interaction steps. In this paper, we propose a novel, unsupervised, and sample-efficient framework to achieve imitation learning from videos (ILV), named Behavior Cloning from Videos via Latent Representations (BCV-LR). BCV-LR extracts action-related latent features from high-dimensional video inputs through self-supervised tasks, and then leverages a dynamics-based unsupervised objective to predict latent actions between consecutive frames. The pre-trained latent actions are fine-tuned and efficiently aligned to the real action space online (with collected interactions) for policy behavior cloning. The cloned policy in turn enriches the agent experience for further latent action finetuning, resulting in an iterative policy improvement that is highly sample-efficient. We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control. BCV-LR enables effective (even expert-level on some tasks) policy performance with only a few interactions, surpassing state-of-the-art ILV baselines and reinforcement learning methods (provided with environmental rewards) in terms of sample efficiency across 24/28 tasks. To the best of our knowledge, this work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need to access any other expert supervision.

AAMAS Conference 2024 Conference Paper

Aligning Credit for Multi-Agent Cooperation via Model-based Counterfactual Imagination

Jiajun Chai
Yuqian Fu
Dongbin Zhao
Yuanheng Zhu

Recent years have witnessed considerable progress in model-based reinforcement learning research. Inspired by the significant improvement in sample efficiency, researchers have explored its application in multi-agent scenarios to mitigate the huge demands in training data of multi-agent reinforcement learning (MARL) approaches. However, existing methods retain the training framework designed for single-agent settings, resulting in inadequate promotion of multi-agent cooperation. In this work, we propose a novel model-based MARL method called Multi-Agent Counterfactual Dreamer (MACD). MACD introduces a centralized imagination with decentralized execution (CIDE) framework to generate higher-quality pseudo data for policy learning, thus further improving the algorithm’s sample efficiency. Moreover, we address the credit assignment and non-stationary challenges by performing an additional counterfactual trajectory based on the learned world model. We provide a theoretical proof that this counterfactual policy update rule maximizes the multi-agent learning objective. Empirical studies validate the superiority of our method in terms of sample efficiency, training stability, and final cooperation performance when compared with several state-of-the-art model-free and model-based MARL algorithms. Ablation studies and visualization demonstration further underscore the significance of both the CIDE framework and the counterfactual module in our approach.

AAMAS Conference 2024 Conference Paper

Boosting Continuous Control with Consistency Policy

Yuhui Chen
Haoran Li
Dongbin Zhao

Due to its training stability and strong expression, the diffusion model has attracted considerable attention in offline reinforcement learning. However, several challenges have also come with it: 1) The demand for a large number of diffusion steps makes the diffusionmodel-based methods time inefficient and limits their applications in real-time control; 2) How to achieve policy improvement with accurate guidance for diffusion model-based policy is still an open problem. Inspired by the consistency model, we propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL), which derives action from noise by a single step. By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance when updating diffusion model-based policy with the learned Q-function. We demonstrate that CPQL can achieve policy improvement with accurate guidance for offline reinforcement learning, and can be seamlessly extended for online RL tasks. Experimental results indicate that CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL. Code is available at https: //github. com/cccedric/cpql.

NeurIPS Conference 2024 Conference Paper

Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization

Haoran Li
Zhennan Jiang
Yuhui Chen
Dongbin Zhao

With high-dimensional state spaces, visual reinforcement learning (RL) faces significant challenges in exploitation and exploration, resulting in low sample efficiency and training stability. As a time-efficient diffusion model, although consistency models have been validated in online state-based RL, it is still an open question whether it can be extended to visual RL. In this paper, we investigate the impact of non-stationary distribution and the actor-critic framework on consistency policy in online RL, and find that consistency policy was unstable during the training, especially in visual RL with the high-dimensional state space. To this end, we suggest sample-based entropy regularization to stabilize the policy training, and propose a consistency policy with prioritized proximal experience regularization (CP3ER) to improve sample efficiency. CP3ER achieves new state-of-the-art (SOTA) performance in 21 tasks across DeepMind control suite and Meta-world. To our knowledge, CP3ER is the first method to apply diffusion/consistency models to visual RL and demonstrates the potential of consistency models in visual RL.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement

Zhi Wang
Li Zhang
Wenhao Wu
Yuanheng Zhu
Dongbin Zhao
Chunlin Chen

A longstanding goal of artificial general intelligence is highly capable generalists that can learn from diverse experiences and generalize to unseen tasks. The language and vision communities have seen remarkable progress toward this trend by scaling up transformer-based models trained on massive datasets, while reinforcement learning (RL) agents still suffer from poor generalization capacity under such paradigms. To tackle this challenge, we propose Meta Decision Transformer (Meta-DT), which leverages the sequential modeling ability of the transformer architecture and robust task representation learning via world model disentanglement to achieve efficient generalization in offline meta-RL. We pretrain a context-aware world model to learn a compact task representation, and inject it as a contextual condition to the causal transformer to guide task-oriented sequence generation. Then, we subtly utilize history trajectories generated by the meta-policy as a self-guided prompt to exploit the architectural inductive bias. We select the trajectory segment that yields the largest prediction error on the pretrained world model to construct the prompt, aiming to encode task-specific information complementary to the world model maximally. Notably, the proposed framework eliminates the requirement of any expert demonstration or domain knowledge at test time. Experimental results on MuJoCo and Meta-World benchmarks across various dataset types show that Meta-DT exhibits superior few and zero-shot generalization capacity compared to strong baselines while being more practical with fewer prerequisites. Our code is available at https: //github. com/NJU-RL/Meta-DT.

PDF Details DOI

ICRA Conference 2023 Conference Paper

STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation

Yupeng Zheng
Chengliang Zhong
Pengfei Li 0007
Huan-ang Gao
Yuhang Zheng 0004
Bu Jin
Ling Wang 0001
Hao Zhao 0002

Self-supervised depth estimation draws a lot of attention recently as it can promote the 3D sensing capa-bilities of self-driving vehicles. However, it intrinsically relies upon the photometric consistency assumption, which hardly holds during nighttime. Although various supervised night-time image enhancement methods have been proposed, their generalization performance in challenging driving scenarios is not satisfactory. To this end, we propose the first method that jointly learns a nighttime image enhancer and a depth estimator, without using ground truth for either task. Our method tightly entangles two self-supervised tasks using a newly proposed uncertain pixel masking strategy. This strategy originates from the observation that nighttime images not only suffer from underexposed regions but also from overexposed regions. By fitting a bridge-shaped curve to the illumination map distribution, both regions are suppressed and two tasks are bridged naturally. We benchmark the method on two established datasets: nuScenes and RobotCar and demonstrate state-of-the-art performance on both of them. Detailed ablations also reveal the mechanism of our proposal. Last but not least, to mitigate the problem of sparse ground truth of existing datasets, we provide a new photo-realistically enhanced nighttime dataset based upon CARLA. It brings meaningful new challenges to the community. Codes, data, and models are available at https://github.com/ucaszyp/STEPS.

IROS Conference 2009 Conference Paper

Fuzzy logic based adjustment control of a cable-driven auto-leveling parallel robot

Yi Yu
Jianqiang Yi
Chengdong Li
Dongbin Zhao
Jianhong Zhang

To solve the level-adjusting and force-tuning problems of high accurate and costly payloads when loading and unloading, a cable-driven auto-leveling parallel robot is developed. A hierarchical fuzzy controller, which has the ability to deal with the rule explosion problem, is proposed in this paper. After a brief introduction of the architecture of the closed-loop control system for the cable-driven auto-leveling parallel robot, the construction of the hierarchical fuzzy controller is set up, in which the force offsets of the four cables and the angle deviations of the two diagonal inclinations are chosen as input variables, and the output variables are the position changes of the four linear motion units. The hierarchical fuzzy controller contains two layers - the low level layer which generates two outputs for leveling adjustment and force tuning, and the high level layer which is used to coordinate the two outputs from the low level layer. Experimental results have demonstrated that the hierarchical fuzzy controller can achieve the control objectives with high regulation accuracy and short adjusting time, and can be easily applied to practical systems.

ICRA Conference 2008 Conference Paper

Control of a class of under-actuated systems with saturation using hierarchical sliding mode

Dianwei Qian
Jianqiang Yi
Dongbin Zhao

This paper presents a control scheme of a class of under-actuated systems with saturation using hierarchical sliding mode. This class with a single input and multiple outputs is made up of several subsystems. Based on this physical structure, the hierarchical structure of the sliding mode surfaces is developed as follows. The sliding surface of every subsystem is defined. Then the sliding surface of one subsystem is selected as the first layer sliding surface. The first layer sliding surface is used to construct the second layer sliding surface with the sliding surface of another subsystem. This process continues till all the subsystem sliding surfaces are included. The hierarchical sliding mode control law is deduced by using Lyapunov theorem. On account of saturation nonlinearity of the single input, asymptotic stability of the control system is proven by nonlinear small gain theorem. Parameter ranges of the subsystem sliding surfaces are also given. In practice, simulation and experimental results show the validity of this control method.

IS Journal 2008 Journal Article

DynaCAS: Computational Experiments and Decision Support for ITS

Nan Zhang
Fei-Yue Wang
Fenghua Zhu
Dongbin Zhao
Shuming Tang

Accurate, reliable, and timely traffic information is critical for deployment and operation of intelligent transportation systems (ITSs). Traffic forecasting for travelers and traffic operators should become at least as useful and convenient as weather reports. In the US, the Federal Highway Administration (FHWA) has envisioned a real-time traffic estimation and prediction system (TrEPS) as an ITS support platform that resides at traffic management centers (TMCs) for dynamic route assignment (DRA) and other transportation operations.

ICRA Conference 2008 Conference Paper

Trajectory tracking control of omnidirecitonal wheeled mobile manipulators: Robust neural network based sliding mode approach

Dong Xu
Dongbin Zhao
Jianqiang Yi
Xiang-min Tan
Zonghai Chen

This paper focuses on developing a robust neural network (NN) based sliding mode controller (NNSMC) to solve the trajectory tracking problem of a redundantly-actuated omnidirectional mobile manipulator. The SMC is designed to be robust to disturbances assuring the stability of the system. The NN is used to identify the unstructured uncertainty of system dynamics. The stability of the closed-loop system, the convergence of the NN weight-updating process, and the boundedness of the NN weight estimation errors are all strictly guaranteed. Through theories analysis, we know the controller is also capable of disturbance-rejection in the presence of time varying disturbances. Finally, simulation results demonstrate the proposed NNSMC approach can guarantee the whole system’s convergence to the desired manifold with prescribed performance.

IROS Conference 2007 Conference Paper

Motion regulation of redundantly actuated omni-directional Wheeled Mobile Robots with internal force control

Dongbin Zhao
Jianqiang Yi
Xuyue Deng

Because of the complexity of the mechanisms of redundantly actuated omni-directional Wheeled Mobile Robots (WMR), its motion regulation is a challenging problem, especially for consideration of the interaction force between the redundantly actuated wheels. The interaction force can be decomposed into motion-induced force and internal force, which are orthogonal between each other. Only the motion-induced force contributes to the motion of the robot, while the internal force abrades the wheels components, and causes the reduction of their life span. So the internal force should be eliminated or minimized. In this paper, kinematic model and dynamic model of redundantly actuated omni-directional WMR considering the interaction force is first established. A proportional differential plus motion regulator is presented. An integral feedback internal force controller is applied to minimize the internal force. Simulation results verify the effectiveness of the proposed control scheme. The robot is regulated successfully, and the internal force is reduced efficiently.

IROS Conference 2007 Conference Paper

Robust adaptive tracking control of omnidirecitonal wheeled mobile manipulators

Dong Xu
Dongbin Zhao
Jianqiang Yi
Xiang-min Tan

This paper addresses the trajectory tracking problem for an redundantly-actuated omnidirectional mobile manipulator system with uncertainties and disturbances. The proposed algorithm is robust adaptive control strategy and the parameter estimates are tuned online. First, for designing controller, the conservative upper-bounded function of dynamic model of omnidirectional mobile manipulator system is derived based on the dynamic structure properties. Then, a robust adaptive control scheme is presented to ensure trajectory tracking effect of this closed-loop system. The asymptotical stability is verified a Lyapunov method. Finally, simulation examples are given to demonstrate the proposed approach can guarantee the whole system converge to the desired manifold with prescribed performance.

ICRA Conference 2007 Conference Paper

Robust Control Using Sliding Mode for a Class of Under-Actuated Systems With Mismatched Uncertainties

Dianwei Qian
Jianqiang Yi
Dongbin Zhao

Based on the methodology of sliding mode, this paper presents a robust controller for a class of under-actuated systems with mismatched uncertainties. Such a system consists of a nominal system and the mismatched uncertainties. The structural characteristic of the nominal system is that it is made up of several subsystems. Based on this characteristic, the hierarchical structure of the sliding mode surfaces is designed for the nominal system as follows. Firstly, the nominal system is divided into several subsystems and the sliding mode surface of every subsystem is defined. Secondly, the sliding mode surface of one subsystem is selected as the first layer sliding mode surface. The first layer sliding mode surface is then to construct the second layer sliding mode surface with the sliding mode surface of another subsystem. This process continues till the sliding mode surfaces of all the subsystems are included. For dealing with the mismatched uncertainties, a lumped sliding mode compensator is designed at the last layer sliding mode surface. The asymptotic stability of every layer sliding mode surface and the sliding mode surface of each subsystem is proven theoretically by Barbalat's lemma. Simulation results show the validity of this robust control method through stabilization control of a double inverted pendulums system with mismatched uncertainties.

IROS Conference 2006 Conference Paper

Hierarchical Sliding Mode Control for Series Double Inverted Pendulums System

Dianwei Qian
Jianqiang Yi
Dongbin Zhao
Yinxing Hao

This paper proposes a hierarchical sliding mode controller for series double inverted pendulums system. This provides a simple method to control a class of under-actuated systems with three subsystems by sliding mode control. Firstly, the given system is divided into three subsystems according to its structure characteristic. Then, the 1 st -level sliding mode surface is defined for every subsystem and the 2 nd -level sliding mode surface is constituted by them. Based on the two levels structure, the equivalent control of each subsystem is deduced and the total control law is derived by the Lyapunov stability theorem. The asymptotical stability of the entire sliding mode surfaces is proved theoretically. Finally, simulation results show the validity of this control strategy. And the influence of the controller parameter changes for the performances is also discussed

IROS Conference 2005 Conference Paper

Cascade sliding-mode controller for large-scale underactuated systems

Jianqiang Yi
Wei Wang 0115
Dongbin Zhao
Xiaojing Liu

On the basis of sliding mode control, a new cascade sliding-mode controller (CSMC) for a class of large-scale underactuated systems is proposed. The large-scale underactuated systems include several subsystems. Firstly, two states are chosen to construct the first-layer sliding surface. Secondly, the first-layer sliding surface and one of the left states are used to construct the second-layer sliding surface. This process continues till the last-layer sliding surface is obtained. By theoretical analysis, the cascade sliding-mode controller is proved to be globally stable in the sense that all signals involved are bounded. The simulation results show the validity of this method.

IROS Conference 2005 Conference Paper

Double layer sliding mode control for second-order underactuated mechanical systems

Wei Wang 0115
Jianqiang Yi
Dongbin Zhao
Xiaojing Liu

A new stable sliding mode control method for a class of underactuated mechanical systems is proposed in this paper. The controller has the double-layer structure. Firstly, the system states are divided into several different subsystems. For each of these subsystems, a first-layer sliding plane is constructed. From these first-layer sliding planes, then we further construct a second-layer sliding plane. By analyzing the features of the mathematical model of the underactuated mechanical systems, we derive the sliding-mode control law and indicate the ranges of the controller parameters. Using Lyapunov law, the paper proves the stability of all the sliding planes theoretically. The simulation results show the validity of this method for this class of underactuated mechanical systems.

ICRA Conference 2005 Conference Paper

Pose Estimation and Structure Recovery from Point Pairs

Zhiguang Zhong
Jianqiang Yi
Dongbin Zhao

This paper presents a new feature point pairs based technique for object pose estimation and structure recovery from a single view. It first estimates rotational matrix independently, then computes translation vector and recovers the 3D structure of the object directly. Linear and nonlinear strategies are presented to estimate the rotational matrix. One is for small rotational motion and the other is used to estimate large rotational parameters. When the nonlinear technique is applied, its initial guesses are given automatically by the proposed linear estimation method. On the other hand, the presented structure recovery method is not sensitive to the rotational matrix estimation results. The proposed method is applicable to three, four or more feature points and has no constraints, such as collinear or coplanar, on their relative positions. As the number of feature points increases, the estimation results are improved while the computation cost is almost unchanged. Many experiments are performed on synthetic data and real images to demonstrate the presented technique.

IROS Conference 2005 Conference Paper

Tracking control of mobile manipulator with dynamical uncertainties

Zuoshi Song
Dongbin Zhao
Jianqiang Yi
Xinchun Li

Tracking control problem of mobile manipulators with dynamical uncertainties is addressed in this paper. The controller is designed based on model of mobile manipulators consisting of two cascaded subsystems: a chained-like kinematical model without uncertainties and a dynamical model with uncertainties. The proposed control law can ensure that full states of closed-loop system can track given trajectories in presence of dynamical uncertainties. A globally asymptotic stability is obtained in Lyapunov sense. Simulation studies show feasibility and effectiveness of the proposed approach.

ICRA Conference 2004 Conference Paper

Passive Adaptive Grasp Multi-fingered Humanoid Robot Hand with High Under-actuated Function

Wenzeng Zhang
Qiang Chen 0009
Zhenguo Sun
Dongbin Zhao

This paper proposed a design idea of a novel under-actuated finger mechanism, and designed the finger mechanism. The finger has no actuator in itself, is only driven by the other finger joints and object grasped. The finger is similar to a human finger and can be easily arranged in series to realize a finger with super under-actuation and high integration. It can be mounted in humanoid robot hand to make the hand obtain more DOFs with less actuators, and good grasping function of shape adaptation, decrease the requirement of control system. This paper analyzed the relationship between the grasping force of the finger and its design parameters, proposed the design principle of structure optimization of the finger. Based on the finger, a multi-fingered humanoid robot hand: TH-2 Hand has been designed. TH-2 Hand has many excellent features: high personification, super under-actuation and be very compact, easy to real-time control, small volume, light in weight, strong grasping function, etc.

ICRA Conference 2003 Conference Paper

Under-actuated passive adaptive grasp humanoid robot hand with control of grasping force

Wenzeng Zhang
Qiang Chen 0009
Zhenguo Sun
Dongbin Zhao

Conventional dexterous hands have too many DOFs, their driver systems are too big to be installed in a humanoid robot arm, and their controls are tool complex. This paper develops an under-actuated passive adaptive grasp humanoid robot hand named TH-1 hand with control of grasping force. With humanoid appearance and size, TH-1 hand is light, fewer DOFs, and can be easily controlled. Its motors and driver circuit boards are embedded in itself. These features make it fit to be installed in a humanoid robot arm. In addition, for stably grasping operation, a mechanical finger with control of grasping force is designed and applied in TH-1 hand's index. To get more DOFs with fewer drivers, a novel under-actuated passive adaptive grasp mechanical finger is design and applied in TH-1 hand's thumb.