Temporal logic motion control using actor-critic methods

In this paper, we consider the problem of deploying a robot from a specification given as a temporal logic statement about some properties satisfied by the regions of a large, partitioned environment. We assume that the robot has noisy sensors and actuators and model its motion through the regions of the environment as a Markov Decision Process (MDP). The robot control problem becomes finding the control policy maximizing the probability of satisfying the temporal logic task on the MDP. For a large environment, obtaining transition probabilities for each state-action pair, as well as solving the necessary optimization problem for the optimal policy are usually not computationally feasible. To address these issues, we propose an approximate dynamic programming framework based on a least-square temporal difference learning method of the actor-critic type. This framework operates on sample paths of the robot and optimizes a randomized control policy with respect to a small set of parameters. The transition probabilities are obtained only when needed. Hardware-in-the-loop simulations confirm that convergence of the parameters translates to an approximately optimal policy.

Authors

• Xu Chu Ding
• Jing Wang 0044
• Morteza Lahijanian
• Ioannis Ch. Paschalidis
• Calin Belta

Keywords

• Computational modeling
• Robot sensing systems
• Materials requirements planning
• Manganese
• Automata
• Markov processes
• Policy Gradient Method
• Temporal Logic
• Actuator
• Transition Probabilities
• Dynamic Programming
• Temporal Differences
• Robot Control
• Markov Decision Process
• Regional Environment
• Sample Paths
• State-action Pair
• Approximate Dynamic Programming
• Temporal Difference Learning
• Specific Tasks
• Sequence Of Actions
• Local Optimum
• Adjacent Regions
• Robot Motion
• Robot Model
• Laser Ranging
• Formal Synthesis
• Actor-critic Algorithm
• Motion Primitives
• Markov Decision Process Model