General non-linear Bellman equations

We consider a general class of non-linear Bellman equations. This opens up a design space of algorithms that have interesting properties. This has two potential advantages. First, we can perhaps better model natural phenomena. For instance, hyperbolic discounting has been proposed as a mathematical model that matches human and animal data well, and can therefore be used to explain preference orderings. We present a different mathematical model that matches the same data, but that makes very different predictions under other circumstances. Second, the larger design space can perhaps lead to algorithms that perform bet- ter, similarly to how discount factors are often used in practice even when the true objective is undiscounted. We show that many of the resulting Bellman operators still converge to a fixed point, and therefore that the resulting algorithms are reasonable.

Authors

• Hado van Hasselt DeepMind
• John Quan
• Matteo Hessel
• Diana Borsa
• Andre Barreto

Keywords

No keywords are indexed for this paper.