Goal-driven dimensionality reduction for reinforcement learning

Defining a state representation on which optimal control can perform well is a tedious but crucial process. It typically requires expert knowledge, does not generalize straightforwardly over different tasks and strongly influences the quality of the learned controller. In this paper, we present an autonomous feature construction method for learning low-dimensional manifolds of goal-relevant features jointly with an optimal controller using reinforcement learning. Our method combines information-theoretic algorithms with principal component analysis to performs a return-weighted reduction of the state representation. The method does not require any preprocessing of the data, does not assume strong restrictions on the state representation, and substantially improves the performance of learning by reducing the number of samples required. We show that our method can learn high quality controller in redundant spaces, even from pixels, and outperforms both classical and state-of-the-art deep learning approaches.

Authors

• Simone Parisi
• Simon Ramstedt
• Jan Peters 0001

Keywords

• Aerospace electronics
• Learning (artificial intelligence)
• Trajectory
• Approximation algorithms
• Space exploration
• Dimensionality Reduction
• Optimal Control
• Deep Learning Approaches
• State Representation
• Feature Construction
• Optimization Problem
• Deep Network
• Value Function
• Maximum Likelihood Estimation
• Remainder Of This Paper
• State Space
• Control Parameters
• Lagrange Multiplier
• Kullback-Leibler
• Vanilla
• Reward Function
• Markov Decision Process
• Low-dimensional Representation
• Reinforcement Learning Algorithm
• Update Strategy
• Goal Of The Agent
• Conditional Mutual Information
• Value Function Approximation
• Reference Distribution
• Environmental Boundaries
• Reinforcement Learning Task