Author name cluster

Pascal Klink

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers

2 author rows

ICLR Conference 2024 Conference Paper

Domain Randomization via Entropy Maximization

Gabriele Tiboni
Pascal Klink
Jan Peters 0001
Tatiana Tommasi
Carlo D'Eramo
Georgia Chalvatzaki

Varying dynamics parameters in simulation is a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's behavior but notoriously leads to overly conservative policies when randomizing excessively. In this paper, we propose a novel approach to address sim-to-real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndomization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while retaining generalization capabilities. In achieving this, DORAEMON gradually increases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON in obtaining highly adaptive and generalizable policies, i.e. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. Notably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.

Details

ICLR Conference 2022 Conference Paper

Boosted Curriculum Reinforcement Learning

Pascal Klink
Carlo D'Eramo
Jan Peters 0001
Joni Pajarinen

Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difficulty. However, finding an exact way of reusing action-values in this setting is still a poorly understood problem. In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task. This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the benefit of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented. This procedure allows reusing previous action-values while promoting expressiveness of the action-value function. We theoretically study BCRL as an approximate value iteration algorithm, discussing advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function. Finally, we provide detailed empirical evidence of the benefits of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.

Details

EWRL Workshop 2022 Workshop Paper

Curriculum Reinforcement Learning via Constrained Optimal Transport

Pascal Klink
Haoyi Yang
Carlo D’Eramo
Jan Peters
Joni Pajarinen

Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.

PDF Details

ICML Conference 2022 Conference Paper

Curriculum Reinforcement Learning via Constrained Optimal Transport

Pascal Klink
Haoyi Yang
Carlo D'Eramo
Jan Peters 0001
Joni Pajarinen

Details

JMLR Journal 2021 Journal Article

A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning

Pascal Klink
Hany Abdulsamad
Boris Belousov
Carlo D'Eramo
Jan Peters
Joni Pajarinen

Across machine learning, the use of curricula has shown strong empirical potential to improve learning from data by avoiding local optima of training objectives. For reinforcement learning (RL), curricula are especially interesting, as the underlying optimization has a strong tendency to get stuck in local optima due to the exploration-exploitation trade-off. Recently, a number of approaches for an automatic generation of curricula for RL have been shown to increase performance while requiring less expert knowledge compared to manually designed curricula. However, these approaches are seldomly investigated from a theoretical perspective, preventing a deeper understanding of their mechanics. In this paper, we present an approach for automated curriculum generation in RL with a clear theoretical underpinning. More precisely, we formalize the well-known self-paced learning paradigm as inducing a distribution over training tasks, which trades off between task complexity and the objective to match a desired task distribution. Experiments show that training on this induced distribution helps to avoid poor local optima across RL algorithms in different tasks with uninformative rewards and challenging exploration requirements. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2021. ( edit, beta )

PDF Details

ICRA Conference 2021 Conference Paper

A Variational Infinite Mixture for Probabilistic Inverse Dynamics Learning

Hany Abdulsamad
Peter Nickl
Pascal Klink
Jan Peters 0001

Probabilistic regression techniques in control and robotics applications have to fulfill different criteria of data-driven adaptability, computational efficiency, scalability to high dimensions, and the capacity to deal with different modalities in the data. Classical regressors usually fulfill only a subset of these properties. In this work, we extend seminal work on Bayesian nonparametric mixtures and derive an efficient variational Bayes inference technique for infinite mixtures of probabilistic local polynomial models with well-calibrated certainty quantification. We highlight the model’s power in combining data-driven complexity adaptation, fast prediction, and the ability to deal with discontinuous functions and heteroscedastic noise. We benchmark this technique on a range of large real-world inverse dynamics datasets, showing that the infinite mixture formulation is competitive with classical Local Learning methods and regularizes model complexity by adapting the number of components based on data and without relying on heuristics. Moreover, to showcase the practicality of the approach, we use the learned models for online inverse dynamics control of a Barrett-WAM manipulator, significantly improving the trajectory tracking performance.

Details

IJCAI Conference 2020 Conference Paper

Generalized Mean Estimation in Monte-Carlo Tree Search

Tuan Dam
Pascal Klink
Carlo D'Eramo
Jan Peters
Joni Pajarinen

We consider Monte-Carlo Tree Search (MCTS) applied to Markov Decision Processes (MDPs) and Partially Observable MDPs (POMDPs), and the well-known Upper Confidence bound for Trees (UCT) algorithm. In UCT, a tree with nodes (states) and edges (actions) is incrementally built by the expansion of nodes, and the values of nodes are updated through a backup strategy based on the average value of child nodes. However, it has been shown that with enough samples the maximum operator yields more accurate node value estimates than averaging. Instead of settling for one of these value estimates, we go a step further proposing a novel backup strategy which uses the power mean operator, which computes a value between the average and maximum value. We call our new approach Power-UCT, and argue how the use of the power mean operator helps to speed up the learning in MCTS. We theoretically analyze our method providing guarantees of convergence to the optimum. Finally, we empirically demonstrate the effectiveness of our method in well-known MDP and POMDP benchmarks, showing significant improvement in performance and convergence speed w. r. t. state of the art algorithms.

PDF Details DOI

NeurIPS Conference 2020 Conference Paper

Self-Paced Deep Reinforcement Learning

Pascal Klink
Carlo D'Eramo
Jan R. Peters
Joni Pajarinen

Curriculum reinforcement learning (CRL) improves the learning speed and stability of an agent by exposing it to a tailored series of tasks throughout learning. Despite empirical successes, an open question in CRL is how to automatically generate a curriculum for a given reinforcement learning (RL) agent, avoiding manual design. In this paper, we propose an answer by interpreting the curriculum generation as an inference problem, where distributions over tasks are progressively learned to approach the target task. This approach leads to an automatic curriculum generation, whose pace is controlled by the agent, with solid theoretical motivation and easily integrated with deep RL algorithms. In the conducted experiments, the curricula generated with the proposed algorithm significantly improve learning performance across several environments and deep RL algorithms, matching or outperforming state-of-the-art existing CRL algorithms.

PDF Details

RLDM Conference 2019 Conference Abstract

Measuring Similarities between Markov Decision Processes

Pascal Klink
Jan Peters

Reinforcement Learning - typically defined as an optimization problem in Markov Decision Pro- cesses (MDPs) - received a lot of attention in recent years, as it allowed to solve more and more complex decision making problems even without any prior knowledge about the problem at hand. However, transfer- ring acquired knowledge between problems - i. e. MDPs - is a topic not addressed by classical Reinforcement Learning approaches. Algorithms that do introduce capabilities of transferring knowledge between different MDPs often require them to be related by a parameter, which can then be utilized by the employed function approximators to inter- and extrapolate for example value functions between different MDPs. While this has been shown to work well if the MDPs behave continuously with respect to their parameterization, this assumption does not need to hold for arbitrary problems. In such cases, the function approximators cannot be expected to adequately generalize over different MDPs. Furthermore, such a problem parameterization may not arise naturally for all kinds of MDPs. Nonetheless, humans are able to reason about similarity of problems even without a parameter that relates them and also recognize aforementioned discontinuities, in which a slight change in the problem formulation requires a drastic change in behavior. Such an under- standing of problem similarity would allow reinforcement learning agents to, just like us humans, reason about the change in a given problem and the required behavior rather than relying on the “black-box” gen- eralization capabilities of function approximators. In this work, we propose a similarity measure between MDPs based on Bisimulation metrics that allows for a more rigorous formalization of MDP similarity. A grid-world experiment shows that such an approach is indeed able to express the similarity of MDPs without additional assumptions and account for abrupt changes in the problem structure.

PDF Details