Autonomous drifting using simulation-aided reinforcement learning

We introduce a framework that combines simple and complex continuous state-action simulators with a real-world robot to efficiently find good control policies, while minimizing the number of samples needed from the physical robot. The framework combines the strengths of various simulation levels by first finding optimal policies in a simple model, and then using that solution to initialize a gradient-based learner in a more complex simulation. The policy and transition dynamics from the complex simulation are in turn used to guide the learning in the physical world. A method is developed for transferring information gathered in the physical world back to the learning agent in the simulation. The new information is used to re-evaluate whether the original simulated policy is still optimal given the updated knowledge from the real-world. This reverse transfer is critical to minimizing samples from the physical world. The new framework is demonstrated on a robotic car learning to perform controlled drifting maneuvers. A video of the car's performance can be found at https://youtu.be/opsmd5yuBF0.

Authors

• Mark Cutler
• Jonathan P. How

Keywords

• Computational modeling
• Mathematical model
• Optimal control
• Robots
• Data models
• Heuristic algorithms
• Approximation algorithms
• Optimal Policy
• Physical World
• Transition Dynamics
• Complex Simulation
• Self-driving
• Reverse Transfer
• Learning Algorithms
• Simulated Data
• Prior Information
• Equations Of Motion
• Real-world Data
• Gaussian Process
• True Function
• Long-term Prediction
• Simulation Domain
• Real Robot
• State-action Pair
• Policy Parameters
• State Action Space
• Policy Search
• Optimal Control Policy
• Expert Demonstrations
• Turn Rate
• Deterministic Simulations
• Model-based Reinforcement Learning
• Learning Process
• Simulation Environment
• Cost Function
• Policy Improvement