Vincent Thomas

AAMAS Conference 2025 Conference Paper

Observer-Aware Probabilistic Planning under Partial Observability

• Salomé Lepers
• Vincent Thomas
• Olivier Buffet

We are interested in planning problems where the agent is aware of the presence of an observer, and where this observer is in a partial observability situation. The agent has to choose its strategy so as to optimize the information transmitted by observations. Building on observer-aware Markov decision processes (OAMDPs), we propose a framework to handle this type of problems and thus formalize properties such as legibility, explicability and predictability. This extension of OAMDPs to partial observability can not only handle more realistic problems, but also permits considering dynamic hidden variables of interest. We discuss theoretical properties of PO-OAMDPs and, experimenting with benchmark problems, we analyze HSVI’s convergence behavior with dedicated initializations and study the resulting strategies.

UAI Conference 2023 Conference Paper

Monte-Carlo Search for an Equilibrium in Dec-POMDPs

• Yang You 0003
• Vincent Thomas
• Francis Colas
• Olivier Buffet

Decentralized partially observable Markov decision processes (Dec-POMDPs) formalize the problem of designing individual controllers for a group of collaborative agents under stochastic dynamics and partial observability. Seeking a global optimum is difficult (NEXP complete), but seeking a Nash equilibrium - each agent policy being a best response to the other agents - is more accessible, and allowed addressing infinite-horizon problems with solutions in the form of finite state controllers. In this paper, we show that this approach can be adapted to cases where only a generative model (a simulator) of the Dec-POMDP is available. This requires relying on a simulation-based POMDP solver to construct an agent’s FSC node by node. A related process is used to heuristically derive initial FSCs. Experiment with benchmarks shows that MC-JESP is competitive with existing Dec-POMDP solvers, even better than many offline methods using explicit models.

ICRA Conference 2023 Conference Paper

Robust Robot Planning for Human-Robot Collaboration

• Yang You 0003
• Vincent Thomas
• Francis Colas
• Rachid Alami 0001
• Olivier Buffet

In human-robot collaboration, the objectives of the human are often unknown to the robot. Moreover, even assuming a known objective, the human behavior is also uncertain. In order to plan a robust robot behavior, a key preliminary question is then: How to derive realistic human behaviors given a known objective? A major issue is that such a human behavior should itself account for the robot behavior, otherwise collaboration cannot happen. In this paper, we rely on Markov decision models, representing the uncertainty over the human objective as a probability distribution over a finite set of objective functions (inducing a distribution over human behaviors). Based on this, we propose two contributions: 1) an approach to automatically generate an uncertain human behavior (a policy) for each given objective function while accounting for possible robot behaviors; and 2) a robot planning algorithm that is robust to the above-mentioned uncertainties and relies on solving a partially observable Markov decision process (POMDP) obtained by reasoning on a distribution over human behaviors. A co-working scenario allows conducting experiments and presenting qualitative and quantitative results to evaluate our approach.

ECAI Conference 2020 Conference Paper

Monte Carlo Information-Oriented Planning

• Vincent Thomas
• Gérémy Hutin
• Olivier Buffet

In this article, we discuss how to solve information-gathering problems expressed as ρ-POMDPs, an extension of Partially Observable Markov Decision Processes (POMDPs) whose reward ρ depends on the belief state. Point-based approaches used for solving POMDPs have been extended to solving ρ-POMDPs as belief MDPs when its reward ρ is convex in B or when it is Lipschitz-continuous. In the present paper, we build on the POMCP algorithm to propose a Monte Carlo Tree Search for ρ-POMDPs, aiming for an efficient on-line planner which can be used for any ρ function. Adaptations are required due to the belief-dependent rewards to (i) propagate more than one state at a time, and (ii) prevent biases in value estimates. An asymptotic convergence proof to Ïţ-optimal values is given when ρ is continuous. Experiments are conducted to analyze the algorithms at hand and show that they outperform myopic approaches.

NeurIPS Conference 2018 Conference Paper

rho-POMDPs have Lipschitz-Continuous epsilon-Optimal Value Functions

• Mathieu Fehr
• Olivier Buffet
• Vincent Thomas
• Jilles Dibangoye

Many state-of-the-art algorithms for solving Partially Observable Markov Decision Processes (POMDPs) rely on turning the problem into a “fully observable” problem—a belief MDP—and exploiting the piece-wise linearity and convexity (PWLC) of the optimal value function in this new state space (the belief simplex ∆). This approach has been extended to solving ρ-POMDPs—i. e. , for information-oriented criteria—when the reward ρ is convex in ∆. General ρ-POMDPs can also be turned into “fully observable” problems, but with no means to exploit the PWLC property. In this paper, we focus on POMDPs and ρ-POMDPs with λ ρ -Lipschitz reward function, and demonstrate that, for finite horizons, the optimal value function is Lipschitz-continuous. Then, value function approximators are proposed for both upper- and lower-bounding the optimal value function, which are shown to provide uniformly improvable bounds. This allows proposing two algorithms derived from HSVI which are empirically evaluated on various benchmark problems.

ECAI Conference 2014 Conference Paper

Simultaneous Tracking and Activity Recognition (STAR) using Advanced Agent-Based Behavioral Simulations

• Arsène Fansi Tchango
• Vincent Thomas
• Olivier Buffet
• Fabien Flacher
• Alain Dutech

Tracking and understanding moving pedestrian behaviors is of major concern for a growing number of applications. Classical approaches either consider both problems separately or treat them simultaneously on the basis of limited contextual graphical models. In this paper, we consider tackling both problems jointly based on richer contextual information issued from agent-based behavioral simulators designed for realistically reproducing human behaviors within complex environments. We focus on the single target case and experimentally show that the proposed approach keeps good performances even in case of long periods of occlusion.

ICML Conference 2012 Conference Paper

Near-Optimal BRL using Optimistic Local Transitions

• Mauricio Araya-López
• Olivier Buffet
• Vincent Thomas

EWRL Workshop 2011 Conference Paper

Active Learning of MDP Models

• Mauricio Araya-López
• Olivier Buffet
• Vincent Thomas
• François Charpillet

Abstract We consider the active learning problem of inferring the transition model of a Markov Decision Process by acting and observing transitions. This is particularly useful when no reward function is a priori defined. Our proposal is to cast the active learning task as a utility maximization problem using Bayesian reinforcement learning with belief-dependent rewards. After presenting three possible performance criteria, we derive from them the belief-dependent rewards to be used in the decision-making process. As computing the optimal Bayesian value function is intractable for large horizons, we use a simple algorithm to approximately solve this optimization problem. Despite the sub-optimality of this technique, we show experimentally that our proposal is efficient in a number of domains.

NeurIPS Conference 2010 Conference Paper

A POMDP Extension with Belief-dependent Rewards

• Mauricio Araya
• Olivier Buffet
• Vincent Thomas
• Françcois Charpillet

Partially Observable Markov Decision Processes (POMDPs) model sequential decision-making problems under uncertainty and partial observability. Unfortunately, some problems cannot be modeled with state-dependent reward functions, e. g. , problems whose objective explicitly implies reducing the uncertainty on the state. To that end, we introduce rho-POMDPs, an extension of POMDPs where the reward function rho depends on the belief state. We show that, under the common assumption that rho is convex, the value function is also convex, what makes it possible to (1) approximate rho arbitrarily well with a piecewise linear and convex (PWLC) function, and (2) use state-of-the-art exact or approximate solving algorithms with limited changes.

AAMAS Conference 2002 Conference Paper

How social spiders inspired an approach for region detection.

• Christine Bourjot
• Vincent Chevrier
• Vincent Thomas