Author name cluster

Patrick Wienhöft

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

AAAI Conference 2025 Conference Paper

Solving Robust Markov Decision Processes: Generic, Reliable, Efficient

Tobias Meggendorfer
Maximilian Weininger
Patrick Wienhöft

Markov decision processes (MDP) are a well-established model for sequential decision-making in the presence of probabilities. In *robust* MDP (RMDP), every action is associated with an *uncertainty set* of probability distributions, modelling that transition probabilities are not known precisely. Based on the known theoretical connection to stochastic games, we provide a framework for solving RMDPs that is generic, reliable, and efficient. It is *generic* both with respect to the model, allowing for a wide range of uncertainty sets, including but not limited to intervals, L1- or L2-balls, and polytopes; and with respect to the objective, including long-run average reward, undiscounted total reward, and stochastic shortest path. It is *reliable*, as our approach not only converges in the limit, but provides precision guarantees at any time during the computation. It is *efficient* because -- in contrast to state-of-the-art approaches -- it avoids explicitly constructing the underlying stochastic game. Consequently, our prototype implementation outperforms existing tools by several orders of magnitude and can solve RMDPs with a million states in under a minute.

PDF Details DOI

IJCAI Conference 2023 Conference Paper

More for Less: Safe Policy Improvement with Stronger Performance Guarantees

Patrick Wienhöft
Marnix Suilen
Thiago D. Simão
Clemens Dubslaff
Christel Baier
Nils Jansen

In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated. State-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance. We present a novel approach to the SPI problem that provides the means to require less data for such guarantees. Specifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI. Our empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.

PDF Details DOI

Highlights Conference 2023 Conference Abstract

Strategy Synthesis in Markov Decision Processes Under Limited Sampling Access

Patrick Wienhöft

A central task in control theory, artificial intelligence, andformal methods is to synthesize reward-maximizing strategies for agentsthat operate in partially unknown environments. In environments modelled by gray-box Markov decision processes (MDPs), the impact of theagents’ actions are known in terms of successor states but not the stochasticsinvolved. In this talk I will present a new strategy synthesis algorithm forgray-box MDPs, via reinforcement learning that utilizes interval MDPs asinternal model. To compete with limited sampling access in reinforcementlearning, the algorithm incorporates two novel concepts, focusingon rapid and successful learning rather than on optimality: lower confidence bound exploration reinforces variantsof already learned practical strategies and action scoping reduces thelearning action space to promising actions. In particular, I will focus on how these methods can be applied inthe context of formal verification of queries asking for theexistence of a strategy under certain constraints. I will illustrate benefits of ouralgorithms by means of a prototypical implementation applied on examplesfrom the AI and formal methods communities and discuss possible extensions. This is joint work with Christel Baier, Clemens Dubslaff, and Stefan J. Kiebel. The full paper has been accepted for publication at the NASA Formal Methods conference 2023. An extended pre-print is avaiable. Contributed talk given by Patrick Wienhöft

Details