A Multitask and Kernel Approach for Learning to Push Objects with a Target-Parameterized Deep Q-Network

Pushing is an essential motor skill involved in several manipulation tasks, and has been an important research topic in robotics. Recent works have shown that Deep Q-Networks (DQNs) can learn pushing policies (when, where to push, and how) to solve manipulation tasks, potentially in synergy with other skills (e. g. grasping). Nevertheless, DQNs often assume a fixed setting and task, which may limit their deployment in practice. Furthermore, they suffer from sparse-gradient backpropagation when the action space is very large, a problem exacerbated by the fact that they are trained to predict state-action values based on a single reward function aggregating several facets of the task, rendering the model training challenging. To address these issues, we propose a multi-head target-parameterized DQN to learn robotic manipulation tasks, in particular pushing policies, and make the following contributions: i) we show that learning to predict different reward and task aspects can be beneficial compared to predicting a single value function where reward factors are not disentangled; ii) we study several alternatives to generalize a policy by encoding the target parameters either into the network layers or visually in the input; iii) we propose a kernelized version of the loss function, allowing to obtain better, faster and more stable training performance. Extensive experiments on simulations validate our design choices, and we show that our architecture learned on simulated data can achieve high performance in a real-robot setup involving a Franka Emika robot arm and unseen objects.

Authors

• Marco Ewerton
• Michael Villamizar
• Julius Jankowski
• Sylvain Calinon
• Jean-Marc Odobez

Keywords

• Training
• Grasping
• Predictive models
• Rendering (computer graphics)
• Manipulators
• Encoding
• Data models
• Deep Q-network
• Kernel Approach
• Value Function
• Motor Skills
• Manipulation Tasks
• Robotic Arm
• Robot Manipulator
• Target Parameters
• Rewarding Aspects
• Deep Learning
• Gaussian Kernel
• Deep Neural Network
• Linear Combination
• Convolutional Layers
• Feature Maps
• Network Training
• Number Of Objects
• Single Network
• Depth Images
• Visual Encoding
• Multi-task Learning
• Validation Loss
• Real Robot
• Policy Learning
• Box Location
• Position Of Box
• Visual Scene
• Backpropagation Algorithm
• Training Loss