Bayes-Adaptive Monte-Carlo Planning and Learning for Goal-Oriented Dialogues

We consider a strategic dialogue task, where the ability to infer the other agent’s goal is critical to the success of the conversational agent. While this problem can be naturally formulated as Bayesian planning, it is known to be a very dif- ficult problem due to its enormous search space consisting of all possible utterances. In this paper, we introduce an ef- ficient Bayes-adaptive planning algorithm for goal-oriented dialogues, which combines RNN-based dialogue generation and MCTS-based Bayesian planning in a novel way, leading to robust decision-making under the uncertainty of the other agent’s goal. We then introduce reinforcement learning for the dialogue agent that uses MCTS as a strong policy improvement operator, casting reinforcement learning as iterative alternation of planning and supervised-learning of self-generated dialogues. In the experiments, we demonstrate that our Bayes-adaptive dialogue planning agent significantly outperforms the state-of-the-art in a negotiation dialogue domain. We also show that reinforcement learning via MCTS further improves end-task performance without diverging from human language.

Authors

• Youngsoo Jang
• JongMin Lee Seoul National University
• Kee-Eung Kim KAIST

Keywords

No keywords are indexed for this paper.