Author name cluster

Ignasi Clavera

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

2 author rows

ICLR Conference 2020 Conference Paper

Model-Augmented Actor-Critic: Backpropagating through Paths

Ignasi Clavera
Yao Fu
Pieter Abbeel

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Details

ICLR Conference 2020 Conference Paper

Sub-policy Adaptation for Hierarchical Reinforcement Learning

Alexander Cong Li
Carlos Florensa
Ignasi Clavera
Pieter Abbeel

Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and videos are available at sites.google.com/view/hippo-rl.

Details

RLDM Conference 2019 Conference Abstract

Deep Reinforcement Learning for Job Scheduling in Computation Graphs on Heterogeneous Platforms

Adam Stooke
Ignasi Clavera
Wenyuan Li
Xin Zhang
Jin Yang

Job scheduling in heterogeneous computing platforms is a challenging real-world problem, per- vasive across a range of system scales. In this work, we use deep reinforcement learning (RL) to optimize scheduling on a parallel computing platform modeled from a real-world networking device. The goal is to complete, as quickly as possible, the computation of a series of jobs with data inter-dependencies, express- ible as a (directed acyclic) job graph. The controller must plan over long horizons to match computation load balancing against the latency of transferring data across the platform. We explore practical aspects of several design spaces: specfication of the (PO)MDP, including reward function with shaping; neural net- work architecture; and learning algorithm. Challenges to learning include: the high-dimensional input and output spaces, partial observability, a sparse figure of merit (graph completion time), and the long problem horizon–in excess of 10, 000 sequential decisions for a realistic job graph. On a high-fidelity simulator, we dramatically outperform two heuristic scheduling schemes while using little-to-no prior domain knowledge. Finally, we discuss future research opportunities in this rich problem, to include reward design, learning algorithm, choices in MDP state specification (e. g. device readout, graph look-ahead), and application of graph-nets. To our knowledge, this work is a unique application of deep RL to a realistic, industrial job scheduling problem, and we believe it has the potential to impact a broad class of computing technologies.

PDF Details

IROS Conference 2017 Conference Paper

Policy transfer via modularity and reward guiding

Ignasi Clavera
David Held
Pieter Abbeel

Non-prehensile manipulation, such as pushing, is an important function for robots to move objects and is sometimes preferred as an alternative to grasping. However, due to unknown frictional forces, pushing has been proven a difficult task for robots. We explore the use of reinforcement learning to train a robot to robustly push an object. In order to deal with the sample complexity of training such a method, we train the pushing policy in simulation and then transfer this policy to the real world. In order to ease the transfer from simulation, we propose to use modularity to separate the learned policy from the raw inputs and outputs; rather than training “end-to-end, ” we decompose our system into modules and train only a subset of these modules in simulation. We further demonstrate that we can incorporate prior knowledge about the task into the state space and the reward function to speed up convergence. Finally, we introduce “reward guiding” to modify the reward function and further reduce the training time. We demonstrate, in both simulation and real-world experiments, that such an approach can be used to reliably push an object from many initial positions and orientations. Videos available at https://goo.gl/B7LtY3.

Details