Discrete off-policy policy gradient using continuous relaxations

Off-Policy policy gradient algorithms are often preferred to on-policy algorithms due to their sample efficiency. Al- though sound off-policy algorithms derived from the policy gradient theorem exist for both discrete and continuous actions, their success in discrete action environments have been limited due to issues arising from off-policy corrections such as importance sampling. This work takes a step in consolidating discrete and continuous off-policy methods by adapting a low-bias, low-variance continuous control method by relaxing a discrete policy into a continuous one. This relaxation allows the action- value function to be differentiable with respect to the discrete policy parameters, and avoids the importance sampling correction typical of off-policy algorithms. Furthermore, the algorithm automatically controls the amount of relaxation, which results in implicit control over exploration. We show that the relaxed algorithm performs comparably to other off-policy algorithms with less hyperparameter tuning.

Authors

• Andre Cianflone
• Zafarali Ahmed
• Riashat Islam
• Joey Bose

Keywords

No keywords are indexed for this paper.