Inference and Learning in Dynamic Decision Networks Using Knowledge Compilation

Decision making under uncertainty in dynamic environments is a fundamental AI problem in which agents need to determine which decisions (or actions) to make at each time step to maximise their expected utility. Dynamic decision networks (DDNs) are an extension of dynamic Bayesian networks with decisions and utilities. DDNs can be used to compactly represent Markov decision processes (MDPs). We propose a novel algorithm called mapl-cirup that leverages knowledge compilation techniques developed for (dynamic) Bayesian networks to perform inference and gradient-based learning in DDNs. Specifically, we knowledge-compile the Bellman update present in DDNs into dynamic decision circuits and evaluate them within an (algebraic) model counting framework. In contrast to other exact symbolic MDP approaches, we obtain differentiable circuits that enable gradient-based parameter learning.

Authors

• Gabriele Venturato KU Leuven
• Vincent Derkinderen KU Leuven
• Pedro Zuidberg Dos Martires Örebro University
• Luc De Raedt KU Leuven Örebro University

Keywords

• ML: Probabilistic Circuits and Graphical Models
• PRS: Planning with Markov Models (MDPs, POMDPs)
• RU: Graphical Models
• RU: Probabilistic Inference
• RU: Sequential Decision Making