Author name cluster

Edouard Klein

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

2 author rows

RLDM Conference 2013 Conference Abstract

Around Inverse Reinforcement Learning and Score-based Classification

Matthieu Geist
Edouard Klein
Bilal Piot
Yann Guermeur

Inverse reinforcement learning (IRL) aims at estimating an unknown reward function optimized by some expert agent from interactions between this expert and the system to be controlled. One of its major application fields is imitation learning, where the goal is to imitate the expert, possibly in situations not encountered before. A classic and simple way to handle this problem is to see it as a classification problem, mapping states to actions. The potential issue with this approach is that classification does not take naturally into account the temporal structure of sequential decision making. Yet, many classification algorithms consist in learning a score function, mapping state-action couples to values, such that the value of the action chosen by the expert is higher than the others. The decision rule of the classifier maximizes the score over actions for a given state. This is curiously reminiscent of the state-action value function in reinforcement learning, and of the associated greedy policy. Based on this simple statement, we propose two IRL algorithms that incorporate the structure of the se- quential decision making problem into some classifier in different ways. The first one, SCIRL (Structured Classification for IRL), starts from the observation that linearly parameterizing a reward function by some features imposes a linear parametrization of the Q-function by a so-called feature expectation. SCIRL simply uses (an estimate of) the expert feature expectation as the basis function of the score function. The second algorithm, CSI (Cascaded Supervised IRL), applies a reversed Bellman equation (expressing the reward as a function of the Q-function) to the score function outputted by any score-based classifier, which reduces to a simple (and generic) regression step. These two algorithms come with theoretical guarantees and perform competitively on toy problems.

PDF Details

NeurIPS Conference 2012 Conference Paper

Inverse Reinforcement Learning through Structured Classification

Edouard Klein
Matthieu Geist
Bilal Piot
Olivier Pietquin

This paper adresses the inverse reinforcement learning (IRL) problem, that is inferring a reward for which a demonstrated expert behavior is optimal. We introduce a new algorithm, SCIRL, whose principle is to use the so-called feature expectation of the expert as the parameterization of the score function of a multi-class classifier. This approach produces a reward function for which the expert policy is provably near-optimal. Contrary to most of existing IRL algorithms, SCIRL does not require solving the direct RL problem. Moreover, with an appropriate heuristic, it can succeed with only trajectories sampled according to the expert behavior. This is illustrated on a car driving simulator.

PDF Details

EWRL Workshop 2011 Conference Paper

Batch, Off-Policy and Model-Free Apprenticeship Learning

Edouard Klein
Matthieu Geist
Olivier Pietquin

Abstract This paper addresses the problem of apprenticeship learning, that is learning control policies from demonstration by an expert. An efficient framework for it is inverse reinforcement learning (IRL). Based on the assumption that the expert maximizes a utility function, IRL aims at learning the underlying reward from example trajectories. Many IRL algorithms assume that the reward function is linearly parameterized and rely on the computation of some associated feature expectations, which is done through Monte Carlo simulation. However, this assumes to have full trajectories for the expert policy as well as at least a generative model for intermediate policies. In this paper, we introduce a temporal difference method, namely LSTD- μ, to compute these feature expectations. This allows extending apprenticeship learning to a batch and off-policy setting.

Details