Author name cluster

Merlijn Krale

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

AAMAS Conference 2026 Conference Paper

Perception-Based Beliefs for POMDPs with Visual Observations

Miriam Schäfers
Merlijn Krale
Thiago D. Simão
Nils Jansen
Maximilian Weininger

Partially observable Markov decision processes (POMDPs) are a principled planning model for sequential decision-making under uncertainty. Yet, real-world problems with high-dimensional observations—such as camera images—remain intractable for traditional belief- and filtering-based solvers. To tackle this problem, we introduce the Perception-based Beliefs for POMDPs framework (PBP), which complements such solvers with a perception model. This model takes the form of an image classifier which maps visual observations to probability distributions over states. PBP incorporates these distributions directly into belief updates, so the underlying solver does not need to reason explicitly over high-dimensional observation spaces. We show that the belief update of PBP coincides with the standard belief update if the image classifier is exact. Moreover, to handle classifier imprecision, we incorporate uncertainty quantification and introduce two methods to adjust the belief update accordingly. We implement PBP using two traditional POMDP solversandempiricallyshowthat(1)itoutperformsexistingend-toend deep RL methods and (2) uncertainty quantification improves robustness of PBP against visual corruption.

PDF

NeurIPS Conference 2025 Conference Paper

On Evaluating Policies for Robust POMDPs

Merlijn Krale
Eline M. Bovy
Maris F. L. Galesloot
Thiago Simão
Nils Jansen

Robust partially observable Markov decision processes (RPOMDPs) model sequential decision-making problems under partial observability, where an agent must be robust against a range of dynamics. RPOMDPs can be viewed as a two-player game between an agent, who selects actions, and nature, who adversarially selects the dynamics. Evaluating an agent policy requires finding an adversarial nature policy, which is computationally challenging. In this paper, we advance the evaluation of agent policies for RPOMDPs in three ways. First, we discuss suitable benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be found by considering only subsets of nature policies, making them easier to solve. We formalize this concept of solvability and construct three benchmarks that are only solvable for expressive sets of nature policies. Second, we describe a new method to evaluate agent policies for RPOMDPs by solving an equivalent MDP. Third, we lift two well-known upper bounds from POMDPs to RPOMDPs, which can be used to efficiently approximate the optimality gap of a policy and serve as baselines. Our experimental evaluation shows that (1) our proposed benchmarks cannot be solved by assuming naive nature policies, (2) our method of evaluating policies is accurate, and (3) the upper bounds provide solid baselines for evaluation.

PDF Details

EWRL Workshop 2025 Workshop Paper

On Evaluating Policies for Robust POMDPs

Merlijn Krale
Eline M. Bovy
Maris F. L. Galesloot
Thiago D. Simão
Nils Jansen

Robust partially observable Markov decision processes (RPOMDPs) model partially observable sequential decision-making problems where an agent must be $\textit{robust}$ against a range of dynamics. RPOMDPs can be viewed as two-player games between an agent, which selects actions, and $\textit{nature}$, which adversarially selects dynamics. Evaluating an agent policy requires finding an adversarial nature policy, which is computationally challenging. In this paper, we advance the evaluation of agent policies for RPOMDPs in three ways. First, we discuss suitable benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be found by considering only subsets of nature policies, making them easier to solve. We formalize this concept of $\textit{solvability}$ and construct three benchmarks that are only solvable for expressive sets of nature policies. Second, we describe a provably sound method to evaluate agent policies for RPOMDPs by solving an equivalent MDP. Third, we lift two well-known POMDP upper value bounds to RPOMDPs, which can be used to efficiently approximate the optimality gap of a policy and serve as baselines. Our experimental evaluation shows that (1) our proposed benchmarks cannot be solved by assuming naive nature policies, (2) our method of evaluating policies is accurate, and (3) the approximations provide solid baselines for evaluation.

PDF

AAMAS Conference 2025 Conference Paper

Tighter Value-Function Approximations for POMDPs

Merlijn Krale
Wietze Koops
Sebastian Junges
Thiago D. Simão
Nils Jansen

Solving partially observable Markov decision processes (POMDPs) typically requires reasoning about the values of exponentially many state beliefs. Towards practical performance, state-of-the-art solvers use value bounds to guide this reasoning. However, sound upper value bounds are often computationally expensive to compute, and there is a tradeoff between the tightness of such bounds and their computational cost. This paper introduces new and provably tighter upper value bounds than the commonly used fast informed bound. Our empirical evaluation shows that, despite their additional computational overhead, the new upper bounds accelerate state-ofthe-art POMDP solvers on a wide range of benchmarks.

PDF

AAAI Conference 2024 Conference Paper

Robust Active Measuring under Model Uncertainty

Merlijn Krale
Thiago D. Simão
Jana Tumova
Nils Jansen

Partial observability and uncertainty are common problems in sequential decision-making that particularly impede the use of formal models such as Markov decision processes (MDPs). However, in practice, agents may be able to employ costly sensors to measure their environment and resolve partial observability by gathering information. Moreover, imprecise transition functions can capture model uncertainty. We combine these concepts and extend MDPs to robust active-measuring MDPs (RAM-MDPs). We present an active-measure heuristic to solve RAM-MDPs efficiently and show that model uncertainty can, counterintuitively, let agents take fewer measurements. We propose a method to counteract this behavior while only incurring a bounded additional cost. We empirically compare our methods to several baselines and show their superior scalability and performance.

PDF Details DOI

ICAPS Conference 2023 Conference Paper

Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring

Merlijn Krale
Thiago D. Simão
Nils Jansen 0001

We study Markov decision processes (MDPs), where agents control when and how they gather information, as formalized by action-contingent noiselessly observable MDPs (ACNO-MPDs). In these models, actions have two components: a control action that influences how the environment changes and a measurement action that affects the agent

Details