Author name cluster

Benoit Boulet

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

TMLR Journal 2025 Journal Article

DRDT3: Diffusion-Refined Decision Test-Time Training Model

Xingshuai Huang
Di Wu
Benoit Boulet

Decision Transformer (DT), a trajectory modelling method, has shown competitive performance compared to traditional offline reinforcement learning (RL) approaches on various classic control tasks. However, it struggles to learn optimal policies from suboptimal, reward-labelled trajectories. In this study, we explore the use of conditional generative modelling to facilitate trajectory stitching given its high-quality data generation ability. Additionally, recent advancements in Recurrent Neural Networks (RNNs) have shown their linear complexity and competitive sequence modelling performance over Transformers. We leverage the Test-Time Training (TTT) layer, an RNN that updates hidden states during testing, to model trajectories in the form of DT. We introduce a unified framework, called Diffusion-Refined Decision TTT (DRDT3), to achieve performance beyond DT models. Specifically, we propose the Decision TTT (DT3) module, which harnesses the sequence modelling strengths of both self-attention and the TTT layer to capture recent contextual information and make coarse action predictions. DRDT3 iteratively refines the coarse action predictions through the generative diffusion model, progressively moving closer to the optimal actions. We further integrate DT3 with the diffusion model using a unified optimization objective. With experiments on multiple tasks in the D4RL benchmark, our DT3 model without diffusion refinement demonstrates improved performance over standard DT, while DRDT3 further achieves superior results compared to state-of-the-art DT-based and offline RL methods.

PDF Details

TMLR Journal 2025 Journal Article

Goal-Conditioned Data Augmentation for Offline Reinforcement Learning

Xingshuai Huang
Di Wu
Benoit Boulet

Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it generally fails to learn well-qualified policies in suboptimal datasets. To address datasets with insufficient optimal demonstrations, we introduce Goal-cOnditioned Data Augmentation (GODA), a novel goal-conditioned diffusion-based method for augmenting samples with higher quality. Leveraging recent advancements in generative modelling, GODA incorporates a novel return-oriented goal condition with various selection mechanisms. Specifically, we introduce a controllable scaling technique to provide enhanced return-based guidance during data sampling. GODA learns a comprehensive distribution representation of the original offline datasets while generating new data with selectively higher-return goals, thereby maximizing the utility of limited optimal demonstrations. Furthermore, we propose a novel adaptive gated conditioning method for processing noisy inputs and conditions, enhancing the capture of goal-oriented guidance. We conduct experiments on the D4RL benchmark and real-world challenges, specifically traffic signal control (TSC) tasks, to demonstrate GODA's effectiveness in enhancing data quality and superior performance compared to state-of-the-art data augmentation methods across various offline RL algorithms.

PDF Details

ICML Conference 2024 Conference Paper

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

Yuwei Fu
Haichao Zhang
Di Wu 0044
Wei Xu
Benoit Boulet

In this work, we investigate how to leverage pre-trained visual-language models (VLM) for online Reinforcement Learning (RL). In particular, we focus on sparse reward tasks with pre-defined textual task descriptions. We first identify the problem of reward misalignment when applying VLM as a reward in RL tasks. To address this issue, we introduce a lightweight fine-tuning method, named Fuzzy VLM reward-aided RL (FuRL), based on reward alignment and relay RL. Specifically, we enhance the performance of SAC/DrQ baseline agents on sparse reward tasks by fine-tuning VLM representations and using relay RL to avoid local minima. Extensive experiments on the Meta-world benchmark tasks demonstrate the efficacy of the proposed method. Code is available at: https: //github. com/fuyw/FuRL.

Details

NeurIPS Conference 2024 Conference Paper

Robot Policy Learning with Temporal Optimal Transport Reward

Yuwei Fu
Haichao Zhang
Di Wu
Wei Xu
Benoit Boulet

Reward specification is one of the most tricky problems in Reinforcement Learning, which usually requires tedious hand engineering in practice. One promising approach to tackle this challenge is to adopt existing expert video demonstrations for policy learning. Some recent work investigates how to learn robot policies from only a single/few expert video demonstrations. For example, reward labeling via Optimal Transport (OT) has been shown to be an effective strategy to generate a proxy reward by measuring the alignment between the robot trajectory and the expert demonstrations. However, previous work mostly overlooks that the OT reward is invariant to temporal order information, which could bring extra noise to the reward signal. To address this issue, in this paper, we introduce the Temporal Optimal Transport (TemporalOT) reward to incorporate temporal order information for learning a more accurate OT-based proxy reward. Extensive experiments on the Meta-world benchmark tasks validate the efficacy of the proposed method. Our code is available at: https: //github. com/fuyw/TemporalOT.

PDF Details DOI

IROS Conference 2024 Conference Paper

Towards Enhanced Fairness and Sample Efficiency in Traffic Signal Control

Xingshuai Huang
Di Wu 0044
Michael Jenkin
Benoit Boulet

Traffic signal control (TSC) has seen substantial advancements through the application of reinforcement learning (RL) algorithms, which have shown remarkable potential in enhancing traffic flow efficiency. These RL-based approaches often surpass traditional rule-based methods, particularly in dynamic traffic environments. However, current RL solutions for TSC predominantly rely on model-free methods, necessitating extensive environmental interactions during training. This requirement can be prohibitively expensive or unfeasible in real-world implementations. Furthermore, existing methods have frequently neglected the issue of fairness in multi-intersection control, resulting in unbalanced congestion across different intersections. To address these challenges, we present FM2Light, a fairness-aware model-based multi-agent RL framework for TSC. Our approach leverages an ensemble of global world models for generating synthetic samples to enhance sample efficiency, thereby mitigating the data-intensive nature of the training process. Additionally, FM2Light incorporates a refined reward structure to promote fairness and improve coordination across multiple intersections. Extensive evaluations conducted in diverse real-world scenarios demonstrate that FM2Light achieves performance comparable to or exceeding that of model-free RL (MFRL) methods, while significantly reducing sample requirements and ensuring more equitable control among multiple agents.

Details

NeurIPS Conference 2022 Conference Paper

A Closer Look at Offline RL Agents

Yuwei Fu
Di Wu
Benoit Boulet

Despite recent advances in the field of Offline Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents. As a result, there remain some gaps in our understandings, i. e. , why is one offline RL agent more performant than another? In this work, we first introduce a set of experiments to evaluate offline RL agents, focusing on three fundamental aspects: representations, value functions and policies. Counterintuitively, we show that a more performant offline RL agent can learn relatively low-quality representations and inaccurate value functions. Furthermore, we showcase that the proposed experiment setups can be effectively used to diagnose the bottleneck of offline RL agents. Inspired by the evaluation results, a novel offline RL algorithm is proposed by a simple modification of IQL and achieves SOTA performance. Finally, we investigate when a learned dynamics model is helpful to model-free offline RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises. Code is available at: https: //github. com/fuyw/RIQL.

PDF Details

AAAI Conference 2022 Conference Paper

Reinforcement Learning Based Dynamic Model Combination for Time Series Forecasting

Yuwei Fu
Di Wu
Benoit Boulet

Time series data appears in many real-world fields such as energy, transportation, communication systems. Accurate modelling and forecasting of time series data can be of significant importance to improve the efficiency of these systems. Extensive research efforts have been taken for time series problems. Different types of approaches, including both statistical-based methods and machine learning-based methods, have been investigated. Among these methods, ensemble learning has shown to be effective and robust. However, it is still an open question that how we should determine weights for base models in the ensemble. Sub-optimal weights may prevent the final model from reaching its full potential. To deal with this challenge, we propose a reinforcement learning (RL) based model combination (RLMC) framework for determining model weights in an ensemble for time series forecasting tasks. By formulating model selection as a sequential decision-making problem, RLMC learns a deterministic policy to output dynamic model weights for non-stationary time series data. RLMC further leverages deep learning to learn hidden features from raw time series data to adapt fast to the changing data distribution. Extensive experiments on multiple real-world datasets have been implemented to showcase the effectiveness of the proposed method.

PDF Details

IROS Conference 2018 Conference Paper

A Sensor-less Catheter Contact Force Estimation Approach in Endovascular Intervention Procedures *

Masoud Razban
Javad Dargahi
Benoit Boulet

Catheter/guidewire manipulation in endovascu-lar intervention procedures are associated with risks of injury on vessel wall and embolization. Determination of catheter/guidewire-vessel interaction contact forces can improve the navigation process safety and efficiency which prevent injuries in both manual and robotic vascular interventions. This study proposes a sensor-less sensing solution to estimate multiple contact point forces at the side of catheter/guidewire exerted on the vasculature. This goal is achieved by using image feedback of catheter-vessel interaction and numerical finite element modeling (FEM). Real-time image processing algorithms are implemented to track interaction contact points on catheter/guidewire. Image-based deflection measurement and contact points tracking data are given to a nonlinear finite element beam model to estimate the forces. The variable equivalent bending modulus of the guidewire is found through a series of three-point-bending tests. To directly measure contact point forces, an experimental platform is prepared which simulates catheter/guidewire-vessel interaction with two, three and four contact points. The effectiveness of the proposed approach is tested in six scenarios in which force estimation accuracy of more than 87. 9% is achieved. The proposed approach can be applied to various types of under-actuated catheter/guidewire in endovascular intervention procedures. This study proves that multiple catheter/guidewire side contact forces can be estimated by using the deflected shape and equivalent bending modulus property without embedding any force sensor.

Details