Arrow Research search

Author name cluster

Anton Raichuk

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers
2 author rows

Possible papers

7

ICML Conference 2022 Conference Paper

Continuous Control with Action Quantization from Demonstrations

  • Robert Dadashi
  • Léonard Hussenot
  • Damien Vincent
  • Sertan Girgin
  • Anton Raichuk
  • Matthieu Geist
  • Olivier Pietquin

In this paper, we propose a novel Reinforcement Learning (RL) framework for problems with continuous action spaces: Action Quantization from Demonstrations (AQuaDem). The proposed approach consists in learning a discretization of continuous action spaces from human demonstrations. This discretization returns a set of plausible actions (in light of the demonstrations) for each input state, thus capturing the priors of the demonstrator and their multimodal behavior. By discretizing the action space, any discrete action deep RL technique can be readily applied to the continuous control problem. Experiments show that the proposed approach outperforms state-of-the-art methods such as SAC in the RL setup, and GAIL in the Imitation Learning setup. We provide a website with interactive videos: https: //google-research. github. io/aquadem/ and make the code available: https: //github. com/google-research/google-research/tree/master/aquadem.

Details

EWRL Workshop 2022 Workshop Paper

Continuous Control with Action Quantization from Demonstrations

  • Robert Dadashi
  • Léonard Hussenot
  • Damien Vincent
  • Sertan Girgin
  • Anton Raichuk
  • Matthieu Geist
  • Olivier Pietquin

In this paper, we propose a novel Reinforcement Learning (RL) framework for problems with continuous action spaces: Action Quantization from Demonstrations (AQuaDem). The proposed approach consists in learning a discretization of continuous action spaces from human demonstrations. This discretization returns a set of plausible actions (in light of the demonstrations) for each input state, thus capturing the priors of the demonstrator and their multimodal behavior. By discretizing the action space, any discrete action deep RL technique can be readily applied to the continuous control problem. Experiments show that the proposed approach outperforms state-of-the-art methods such as SAC in the RL setup, and GAIL in the Imitation Learning setup.

PDF Details

NeurIPS Conference 2021 Conference Paper

Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation

  • Daniel Freeman
  • Erik Frey
  • Anton Raichuk
  • Sertan Girgin
  • Igor Mordatch
  • Olivier Bachem

We present Brax, an open source library for \textbf{r}igid \textbf{b}ody simulation with a focus on performance and parallelism on accelerators, written in JAX. We present results on a suite of tasks inspired by the existing reinforcement learning literature, but remade in our engine. Additionally, we provide reimplementations of PPO, SAC, ES, and direct policy optimization in JAX that compile alongside our environments, allowing the learning algorithm and the environment processing to occur on the same device, and to scale seamlessly on accelerators. Finally, we include notebooks that facilitate training of performant policies on common MuJoCo-like tasks in minutes.

PDF Details

ICML Conference 2021 Conference Paper

Hyperparameter Selection for Imitation Learning

  • Léonard Hussenot
  • Marcin Andrychowicz
  • Damien Vincent
  • Robert Dadashi
  • Anton Raichuk
  • Sabela Ramos
  • Nikola Momchev
  • Sertan Girgin

We address the issue of tuning hyperparameters (HPs) for imitation learning algorithms in the context of continuous-control, when the underlying reward function of the demonstrating expert cannot be observed at any time. The vast literature in imitation learning mostly considers this reward function to be available for HP selection, but this is not a realistic setting. Indeed, would this reward function be available, it could then directly be used for policy training and imitation would not be necessary. To tackle this mostly ignored problem, we propose a number of possible proxies to the external reward. We evaluate them in an extensive empirical study (more than 10’000 agents across 9 environments) and make practical recommendations for selecting HPs. Our results show that while imitation learning algorithms are sensitive to HP choices, it is often possible to select good enough HPs through a proxy to the reward function.

Details

NeurIPS Conference 2021 Conference Paper

What Matters for Adversarial Imitation Learning?

  • Manu Orsini
  • Anton Raichuk
  • Leonard Hussenot
  • Damien Vincent
  • Robert Dadashi
  • Sertan Girgin
  • Matthieu Geist
  • Olivier Bachem

Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies. It is therefore difficult to discuss and understand what choices, among the high-level algorithmic options as well as low-level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning frameworkand investigate their impacts in a large-scale study (>500k trained agents) with both synthetic and human-generated demonstrations. We analyze the key results and highlight the most surprising findings.

PDF Details

ICLR Conference 2021 Conference Paper

What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study

  • Marcin Andrychowicz
  • Anton Raichuk
  • Piotr Stanczyk
  • Manu Orsini
  • Sertan Girgin
  • Raphaël Marinier
  • Léonard Hussenot
  • Matthieu Geist

In recent years, reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``"choices" in a unified on-policy deep actor-critic framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for the training of on-policy deep actor-critic RL agents.

Details

AAAI Conference 2020 Conference Paper

Google Research Football: A Novel Reinforcement Learning Environment

  • Karol Kurach
  • Anton Raichuk
  • Piotr Stańczyk
  • Michał Zając
  • Olivier Bachem
  • Lasse Espeholt
  • Carlos Riquelme
  • Damien Vincent

Recent progress in the field of reinforcement learning has been accelerated by virtual learning environments such as video games, where novel algorithms and ideas can be quickly tested in a safe and reproducible manner. We introduce the Google Research Football Environment, a new reinforcement learning environment where agents are trained to play football in an advanced, physics-based 3D simulator. The resulting environment is challenging, easy to use and customize, and it is available under a permissive open-source license. In addition, it provides support for multiplayer and multi-agent experiments. We propose three full-game scenarios of varying difficulty with the Football Benchmarks and report baseline results for three commonly used reinforcement algorithms (IMPALA, PPO, and Ape-X DQN). We also provide a diverse set of simpler scenarios with the Football Academy and showcase several promising research directions.

PDF Details
v2026.09.13