Arrow Research search

Author name cluster

Jens Kober

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

44 papers
2 author rows

Possible papers

44

EWRL Workshop 2025 Workshop Paper

An Open-Loop Baseline for Reinforcement Learning Locomotion Tasks

  • Antonin Raffin
  • Olivier Sigaud
  • Jens Kober
  • Alin Albu-Schaeffer
  • João Silvério
  • Freek Stulp

In search of a simple baseline for Deep Reinforcement Learning in locomotion tasks, we propose a model-free open-loop strategy. By leveraging prior knowledge and the elegance of simple oscillators to generate periodic joint motions, it achieves respectable performance in five different locomotion environments, with a number of tunable parameters that is a tiny fraction of the thousands typically required by DRL algorithms. We conduct two additional experiments using open-loop oscillators to identify current shortcomings of these algorithms. Our results show that, compared to the baseline, DRL is more prone to performance degradation when exposed to sensor noise or failure. Furthermore, we demonstrate a successful transfer from simulation to reality using an elastic quadruped, where RL fails without randomization or reward engineering. Overall, the proposed baseline and associated experiments highlight the existing limitations of DRL for robotic applications, provide insights on how to address them, and encourage reflection on the costs of complexity and generality.

PDF

TMLR Journal 2025 Journal Article

ASkDAgger: Active Skill-level Data Aggregation for Interactive Imitation Learning

  • Jelle Luijkx
  • Zlatan Ajanović
  • Laura Ferranti
  • Jens Kober

Human teaching effort is a significant bottleneck for the broader applicability of interactive imitation learning. To reduce the number of required queries, existing methods employ active learning to query the human teacher only in uncertain, risky, or novel situations. However, during these queries, the novice’s planned actions are not utilized despite containing valuable information, such as the novice’s capabilities, as well as corresponding uncertainty levels. To this end, we allow the novice to say: “I plan to do this, but I am uncertain.” We introduce the Active Skill-level Data Aggregation (ASkDAgger) framework, which leverages teacher feedback on the novice plan in three key ways: (1) S-Aware Gating (SAG): Adjusts the gating threshold to track sensitivity, specificity, or a minimum success rate; (2) Foresight Interactive Experience Replay (FIER), which recasts valid and relabeled novice action plans into demonstrations; and (3) Prioritized Interactive Experience Replay (PIER), which prioritizes replay based on uncertainty, novice success, and demonstration age. Together, these components balance query frequency with failure incidence, reduce the number of required demonstration annotations, improve generalization, and speed up adaptation to changing domains. We validate the effectiveness of ASkDAgger through language-conditioned manipulation tasks in both simulation and real-world environments. Code, data, and videos are available at https://askdagger.github.io.

PDF Details

ICRA Conference 2025 Conference Paper

ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models

  • Runyu Ma
  • Jelle Luijkx
  • Zlatan Ajanovic
  • Jens Kober

In robot manipulation, Reinforcement Learning (RL) often suffers from low sample efficiency and uncertain convergence, especially in large observation and action spaces. Foundation Models (FMs) offer an alternative, demonstrating promise in zero-shot and few-shot settings. However, they can be unreliable due to limited physical and spatial understanding. We introduce ExploRLLM, a method that combines the strengths of both paradigms. In our approach, FMs improve RL convergence by generating policy code and efficient representations, while a residual RL agent compensates for the FMs' limited physical understanding. We show that Explorllm outperforms both policies derived from FMs and RL baselines in table-top manipulation tasks. Additionally, real-world experiments show that the policies exhibit promising zero-shot sim-to-real transfer. Supplementary material is available at https://explorllm.github.io.

Details

IROS Conference 2025 Conference Paper

Explosive Jumping with Rigid and Articulated Soft Quadrupeds via Example Guided Reinforcement Learning

  • Georgios Apostolides
  • Wei Pan 0004
  • Jens Kober
  • Cosimo Della Santina
  • Jiatao Ding

Achieving controlled jumping behaviour for a quadruped robot is a challenging task, especially when introducing passive compliance in mechanical design. This study addresses this challenge via imitation-based deep reinforcement learning with a progressive training process. To start, we learn the jumping skill by mimicking a coarse jumping example generated by model-based trajectory optimization. Subsequently, we generalize the learned policy to broader situations, including various distances in both forward and lateral directions, and then pursue robust jumping in unknown ground unevenness. In addition, without tuning the reward much, we learn the jumping policy for a quadruped with parallel elasticity. Results show that using the proposed method, i) the robot learns versatile jumps by learning only from a single demonstration, ii) the robot with parallel compliance reduces the landing error by 11. 1%, saves energy cost by 15. 2% and reduces the peak torque by 15. 8%, compared to the rigid robot without parallel elasticity, iii) the robot can perform jumps of variable distances with robustness against ground unevenness (maximal ±4cm height perturbations) using only proprioceptive perception.

Details

ICRA Conference 2025 Conference Paper

Impedance Primitive-Augmented Hierarchical Reinforcement Learning for Sequential Tasks

  • Amin Berjaoui Tahmaz
  • Ravi Prakash 0002
  • Jens Kober

This paper presents an Impedance Primitive-augmented hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework's sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks.

Details

ICLR Conference 2025 Conference Paper

Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation

  • Anish Abhijit Diwan
  • Julen Urain
  • Jens Kober
  • Jan Peters 0001

This paper introduces a new imitation learning framework based on energy-based generative models capable of learning complex, physics-dependent, robot motion policies through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert's motion data distribution and learns smooth, and well-defined representations of the data distribution's energy function using denoising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined in the manifold of policy-generated samples. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algorithms like Adversarial Motion Priors (AMP). Our framework sidesteps the optimisation challenges of adversarial imitation learning techniques and produces results comparable to AMP in several quantitative metrics across multiple imitation settings.

Details

TMLR Journal 2025 Journal Article

Predictable Reinforcement Learning Dynamics through Entropy Rate Minimization

  • Daniel Jarne Ornia
  • Giannis Delimpaltadakis
  • Jens Kober
  • Javier Alonso-Mora

In Reinforcement Learning (RL), agents have no incentive to exhibit predictable trajectories, and are often pushed (through e.g. policy entropy regularisation) to randomise their actions in favor of exploration. This lack of predictability awareness often makes it challenging for other agents and humans to predict an agent's trajectories, possibly triggering unsafe scenarios (e.g. in human-robot interaction). We propose a novel method to induce predictable trajectories in RL agents, termed Predictability-Aware RL (PARL), employing the agent's trajectory entropy rate to quantify predictability. Our method maximizes a linear combination of a standard discounted reward and the negative entropy rate, thus trading off optimality with predictability. We show how the entropy rate can be formally cast as an average reward, how entropy-rate value functions can be estimated from a learned model and incorporate this in policy-gradient algorithms, and demonstrate how this approach produces predictable (near-optimal) policies in tasks inspired by human-robot use-cases.

PDF Details

TMLR Journal 2025 Journal Article

REX: GPU-Accelerated Sim2Real Framework with Delay and Dynamics Estimation

  • Bas van der Heijden
  • Jens Kober
  • Robert Babuska
  • Laura Ferranti

Sim2real, the transfer of control policies from simulation to the real world, is crucial for efficiently solving robotic tasks without the risks associated with real-world learning. However, discrepancies between simulated and real environments, especially due to unmodeled dynamics and latencies, significantly impact the performance of these transferred policies. In this paper, we address the challenges of sim2real transfer caused by latency and asynchronous dynamics in real-world robotic systems. Our approach involves developing a novel framework, REX (Robotic Environments with jaX), that uses a graph-based simulation model to incorporate latency effects while optimizing for parallelization on accelerator hardware. Our framework simulates the asynchronous, hierarchical nature of real-world systems, while simultaneously estimating system dynamics and delays from real-world data and implementing delay compensation strategies to minimize the sim2real gap. We validate our approach on two real-world systems, demonstrating its effectiveness in improving sim2real performance by accurately modeling both system dynamics and delays. Our results show that the proposed framework supports both accelerated simulation and real-time processing, making it valuable for robot learning.

PDF Details

RLC Conference 2024 Conference Paper

An Open-Loop Baseline for Reinforcement Learning Locomotion Tasks

  • Antonin Raffin
  • Olivier Sigaud
  • Jens Kober
  • Alin Albu-Schaeffer
  • João Silvério
  • Freek Stulp

In search of a simple baseline for Deep Reinforcement Learning in locomotion tasks, we propose a model-free open-loop strategy. By leveraging prior knowledge and the elegance of simple oscillators to generate periodic joint motions, it achieves respectable performance in five different locomotion environments, with a number of tunable parameters that is a tiny fraction of the thousands typically required by DRL algorithms. We conduct two additional experiments using open-loop oscillators to identify current shortcomings of these algorithms. Our results show that, compared to the baseline, DRL is more prone to performance degradation when exposed to sensor noise or failure. Furthermore, we demonstrate a successful transfer from simulation to reality using an elastic quadruped, where RL fails without randomization or reward engineering. Overall, the proposed baseline and associated experiments highlight the existing limitations of DRL for robotic applications, provide insights on how to address them, and encourage reflection on the costs of complexity and generality.

PDF Details

RLJ Journal 2024 Journal Article

An Open-Loop Baseline for Reinforcement Learning Locomotion Tasks

  • Antonin Raffin
  • Olivier Sigaud
  • Jens Kober
  • Alin Albu-Schaeffer
  • João Silvério
  • Freek Stulp

In search of a simple baseline for Deep Reinforcement Learning in locomotion tasks, we propose a model-free open-loop strategy. By leveraging prior knowledge and the elegance of simple oscillators to generate periodic joint motions, it achieves respectable performance in five different locomotion environments, with a number of tunable parameters that is a tiny fraction of the thousands typically required by DRL algorithms. We conduct two additional experiments using open-loop oscillators to identify current shortcomings of these algorithms. Our results show that, compared to the baseline, DRL is more prone to performance degradation when exposed to sensor noise or failure. Furthermore, we demonstrate a successful transfer from simulation to reality using an elastic quadruped, where RL fails without randomization or reward engineering. Overall, the proposed baseline and associated experiments highlight the existing limitations of DRL for robotic applications, provide insights on how to address them, and encourage reflection on the costs of complexity and generality.

PDF Details

TMLR Journal 2024 Journal Article

Efficient Parallelized Simulation of Cyber-Physical Systems

  • Bas van der Heijden
  • Laura Ferranti
  • Jens Kober
  • Robert Babuska

Advancements in accelerated physics simulations have greatly reduced training times for reinforcement learning policies, yet the conventional step-by-step agent-simulator interaction undermines simulation accuracy. In the real-world, interactions are asynchronous, with sensing, acting and processing happening simultaneously. Failing to capture this widens the sim2real gap and results in suboptimal real-world performance. In this paper, we address the challenges of simulating realistic asynchronicity and delays within parallelized simulations, crucial to bridging the sim2real gap in complex cyber-physical systems. Our approach efficiently parallelizes cyber-physical system simulations on accelerator hardware, including physics, sensors, actuators, processing components and their asynchronous interactions. We extend existing accelerated physics simulations with latency simulation capabilities by constructing a `supergraph' that encodes all data dependencies across parallelized simulation steps, ensuring accurate simulation. By finding the smallest supergraph, we minimize redundant computation. We validate our approach on two real-world systems and perform an extensive ablation, demonstrating superior performance compared to baseline methods.

PDF Details

IROS Conference 2024 Conference Paper

Learning Multi-Reference Frame Skills from Demonstration with Task-Parameterized Gaussian Processes

  • Mariano Ramírez Montero
  • Giovanni Franzese
  • Jens Kober
  • Cosimo Della Santina

A central challenge in Learning from Demonstration is to generate representations that are adaptable and can generalize to unseen situations. This work proposes to learn such a representation without using task-specific heuristics within the context of multi-reference frame skill learning by superimposing local skills in the global frame. Local policies are first learned by fitting the relative skills with respect to each frame using Gaussian Processes (GPs). Then, another GP, which determines the relevance of each frame for every time step, is trained in a self-supervised manner from a different batch of demonstrations. The uncertainty quantification capability of GPs is exploited to stabilize the local policies and to train the frame relevance in a fully Bayesian way. We validate the method through a dataset of multi-frame tasks generated in simulation and on real-world experiments with a robotic manipulation pick-and-place re-shelving task. We evaluate the performance of our method with two metrics: how close the generated trajectories get to each of the task goals and the deviation between these trajectories and test expert trajectories. According to both of these metrics, the proposed method consistently outperforms the state-of-the-art baseline, Task-Parameterised Gaussian Mixture Model (TPGMM).

Details

IJCAI Conference 2024 Conference Paper

ROME: Robust Multi-Modal Density Estimator

  • Anna Mészáros
  • Julian F. Schumann
  • Javier Alonso-Mora
  • Arkady Zgonnikov
  • Jens Kober

The estimation of probability density functions is a fundamental problem in science and engineering. However, common methods such as kernel density estimation (KDE) have been demonstrated to lack robustness, while more complex methods have not been evaluated in multi-modal estimation problems. In this paper, we present ROME (RObust Multi-modal Estimator), a non-parametric approach for density estimation which addresses the challenge of estimating multi-modal, non-normal, and highly correlated distributions. ROME utilizes clustering to segment a multi-modal set of samples into multiple uni-modal ones and then combines simple KDE estimates obtained for individual clusters in a single multi-modal estimate. We compared our approach to state-of-the-art methods for density estimation as well as ablations of ROME, showing that it not only outperforms established methods but is also more robust to a variety of distributions. Our results demonstrate that ROME can overcome the issues of over-fitting and over-smoothing exhibited by other estimators.

PDF Details DOI

ICRA Conference 2024 Conference Paper

Two-Stage Learning of Highly Dynamic Motions with Rigid and Articulated Soft Quadrupeds

  • Francesco Vezzi
  • Jiatao Ding
  • Antonin Raffin
  • Jens Kober
  • Cosimo Della Santina

Controlled execution of dynamic motions in quadrupedal robots, especially those with articulated soft bodies, presents a unique set of challenges that traditional methods struggle to address efficiently. In this study, we tackle these issues by relying on a simple yet effective two-stage learning framework to generate dynamic motions for quadrupedal robots. First, a gradient-free evolution strategy is employed to discover simply represented control policies, eliminating the need for a predefined reference motion. Then, we refine these policies using deep reinforcement learning. Our approach enables the acquisition of complex motions like pronking and back-flipping, effectively from scratch. Additionally, our method simplifies the traditionally labour-intensive task of reward shaping, boosting the efficiency of the learning process. Importantly, our framework proves particularly effective for articulated soft quadrupeds, whose inherent compliance and adaptability make them ideal for dynamic tasks but also introduce unique control challenges.

Details

EWRL Workshop 2023 Workshop Paper

Prioritizing States with Action Sensitive Return in Experience Replay

  • Alexander Keijzer
  • Bas van der Heijden
  • Jens Kober

Experience replay for off-policy reinforcement learning has been shown to improve sample efficiency and stabilize training. However, typical uniformly sampled replay includes many irrelevant samples for the agent to reach good performance. We introduce Action Sensitive Experience Replay (ASER), a method to prioritize samples in the replay buffer and selectively model parts of the state-space more accurately where choosing sub-optimal actions has a larger effect on the return. We experimentally show that this can make training more sample efficient and that this allows smaller parametric function approximators -- like neural networks with few neurons -- to achieve good performance in environments where they would otherwise struggle.

PDF

ICRA Conference 2023 Conference Paper

Probabilistic Risk Assessment for Chance-Constrained Collision Avoidance in Uncertain Dynamic Environments

  • Khaled A. Mustafa
  • Oscar de Groot
  • Xinwei Wang 0006
  • Jens Kober
  • Javier Alonso-Mora

Balancing safety and efficiency when planning in crowded scenarios with uncertain dynamics is challenging where it is imperative to accomplish the robot's mission without incurring any safety violations. Typically, chance constraints are incorporated into the planning problem to provide probabilistic safety guarantees by imposing an upper bound on the collision probability of the planned trajectory. Yet, this results in an overly conservative behavior on the grounds that the gap between the obtained risk and the specified upper limit is not explicitly restricted. To address this issue, we propose a real-time capable approach to quantify the risk associated with planned trajectories obtained from multiple probabilistic planners, running in parallel, with different upper bounds of the acceptable risk level. Based on the evaluated risk, the least conservative plan is selected provided that its associated risk is below a specified threshold. In such a way, the proposed approach provides probabilistic safety guarantees by attaining a closer bound to the specified risk, while being applicable to generic uncertainties of moving obstacles. We demonstrate the efficiency of our proposed approach, by improving the performance of a state-of-the-art probabilistic planner, in simulations and experiments using a mobile robot in an environment shared with humans.

Details

PRL Workshop 2023 Workshop Paper

Value Function Learning via Prolonged Backward Heuristic Search

  • Zlatan Ajanovic
  • Bakir Lacevic
  • Jens Kober

In practical applications like autonomous robots, we often need to solve similar problems repeatedly (e.g.\ replanning). Existing methods, that improve search performance based on learning from experience in similar previously solved problems, train the heuristics by imitating oracle data. However, such methods rather focus on generating the data with appropriate distribution (e.g.\ by aggregating data online) rather than the computational complexity of generating it. Computational complexity becomes especially limiting for high-dimensional problems. Here, we present a search-inspired method for systematic model exploration that allows us to efficiently generate data and use all explored states for learning the value function -- that can then be employed as heuristic. Our method helps with data distribution as the search typically explores many more states besides the optimal path. The coverage can be improved even further with the Prolonged Search algorithm, which does not stop when a goal is reached, but rather keeps the search running until an extended region around the optimal path is explored. This, in turn, improves both the efficiency and robustness of successive planning. To address the negative effects of using an ML heuristic, we bound it with other heuristics to prevent (significant) overestimating the cost-to-go and ensure bounds on optimality even for non-iid or out-of-domain data. Our approach outperforms existing methods on benchmark problems and shows promising directions for developing efficient and robust search-based planning systems.

PDF Details

IROS Conference 2022 Conference Paper

Automatic Tuning and Selection of Whole-Body Controllers

  • Evelyn D'Elia
  • Jean-Baptiste Mouret
  • Jens Kober
  • Serena Ivaldi

Designing controllers for complex robots such as humanoids is not an easy task. Often, researchers hand-tune controllers, but this is a time-consuming approach that yields a single controller which cannot generalize well to varied tasks. This work presents a method which uses the NSGA-II multi-objective optimization algorithm with various training trajectories to output a diverse Pareto set of well-functioning controller weights and gains. The best of these are shown to also work well on the real Talos robot. The learned Pareto front is then used in a Bayesian optimization (BO) algorithm both as a search space and as a source of prior information in the initial mean estimate. This combined learning approach, leveraging the two optimization methods together, finds a suitable parameter set for a new trajectory within 20 trials and outperforms both BO in the continuous parameter search space and random search along the precomputed Pareto front. The few trials required for this formulation of BO suggest that it could feasibly be applied on the physical robot using a Pareto front generated in simulation.

Details

IROS Conference 2022 Conference Paper

OpenDR: An Open Toolkit for Enabling High Performance, Low Footprint Deep Learning for Robotics

  • Nikolaos Passalis
  • S. Pedrazzi
  • Robert Babuska
  • Wolfram Burgard
  • D. Dias
  • F. Ferro
  • Moncef Gabbouj
  • Ole Green

Existing Deep Learning (DL) frameworks typically do not provide ready-to-use solutions for robotics, where very specific learning, reasoning, and embodiment problems exist. Their relatively steep learning curve and the different methodologies employed by DL compared to traditional approaches, along with the high complexity of DL models, which often leads to the need of employing specialized hardware accelerators, further increase the effort and cost needed to employ DL models in robotics. Also, most of the existing DL methods follow a static inference paradigm, as inherited by the traditional computer vision pipelines, ignoring active perception, which can be employed to actively interact with the environment in order to increase perception accuracy. In this paper, we present the Open Deep Learning Toolkit for Robotics (OpenDR). OpenDR aims at developing an open, non-proprietary, efficient, and modular toolkit that can be easily used by robotics companies and research institutions to efficiently develop and deploy AI and cognition technologies to robotics applications, providing a solid step towards addressing the aforementioned challenges. We also detail the design choices, along with an abstract interface that was created to overcome these challenges. This interface can describe various robotic tasks, spanning beyond traditional DL cognition and inference, as known by existing frameworks, incorporating openness, homogeneity and robotics-oriented perception e. g. , through active perception, as its core design principles.

Details

EAAI Journal 2022 Journal Article

Visually-guided motion planning for autonomous driving from interactive demonstrations

  • Rodrigo Pérez-Dattari
  • Bruno Brito
  • Oscar de Groot
  • Jens Kober
  • Javier Alonso-Mora

The successful integration of autonomous robots in real-world environments strongly depends on their ability to reason from context and take socially acceptable actions. Current autonomous navigation systems mainly rely on geometric information and hard-coded rules to induce safe and socially compliant behaviors. Yet, in unstructured urban scenarios these approaches can become costly and suboptimal. In this paper, we introduce a motion planning framework consisting of two components: a data-driven policy that uses visual inputs and human feedback to generate socially compliant driving behaviors (encoded by high-level decision variables), and a local trajectory optimization method that executes these behaviors (ensuring safety). In particular, we employ Interactive Imitation Learning to jointly train the policy with the local planner, a Model Predictive Controller (MPC), which results in safe and human-like driving behaviors. Our approach is validated in realistic simulated urban scenarios. Qualitative results show the similarity of the learned behaviors with human driving. Furthermore, navigation performance is substantially improved in terms of safety, i. e. , number of collisions, as compared to prior trajectory optimization frameworks, and in terms of data-efficiency as compared to prior learning-based frameworks, broadening the operational domain of MPC to more realistic autonomous driving scenarios.

Details DOI

IROS Conference 2021 Conference Paper

DeepKoCo: Efficient latent planning with a task-relevant Koopman representation

  • Bas van der Heijden
  • Laura Ferranti
  • Jens Kober
  • Robert Babuska

This paper presents DeepKoCo, a novel modelbased agent that learns a latent Koopman representation from images. This representation allows DeepKoCo to plan efficiently using linear control methods, such as linear model predictive control. Compared to traditional agents, DeepKoCo learns taskrelevant dynamics, thanks to the use of a tailored lossy autoencoder network that allows DeepKoCo to learn latent dynamics that reconstruct and predict only observed costs, rather than all observed dynamics. As our results show, DeepKoCo achieves a similar final performance as traditional model-free methods on complex control tasks, while being considerably more robust to distractor dynamics, making the proposed agent more amenable for real-life applications.

Details

IROS Conference 2021 Conference Paper

ILoSA: Interactive Learning of Stiffness and Attractors

  • Giovanni Franzese
  • Anna Mészáros
  • Luka Peternel
  • Jens Kober

Teaching robots how to apply forces according to our preferences is still an open challenge that has to be tackled from multiple engineering perspectives. This paper studies how to learn variable impedance policies where both the Cartesian stiffness and the attractor can be learned from human demonstrations and corrections with a user-friendly interface. The presented framework, named ILoSA, uses Gaussian Processes for policy learning, identifying regions of uncertainty and allowing interactive corrections, stiffness modulation and active disturbance rejection. The experimental evaluation of the framework is carried out on a Franka-Emika Panda in four separate cases with unique force interaction properties: 1) pulling a plug wherein a sudden force discontinuity occurs upon successful removal of the plug, 2) pushing a box where a sustained force is required to keep the robot in motion, 3) wiping a whiteboard in which the force is applied perpendicular to the direction of movement, and 4) inserting a plug to verify the usability for precision-critical tasks in an experimental validation performed with non-expert users.

Details

ICRA Conference 2021 Conference Paper

Imitation Learning with Inconsistent Demonstrations through Uncertainty-based Data Manipulation

  • Peter Valletta
  • Rodrigo Pérez-Dattari
  • Jens Kober

Aleatoric uncertainty estimation, based on the observed training data, is applied for the detection of conflicts in a demonstration data set. The particular focus of this paper is the resolution of conflicting data resulting from scenarios with equivalent action choices, such as obstacle avoidance, path planning or multiple joint configurations. In terms of the estimated uncertainty, the proposed algorithm aims to decrease this otherwise irreducible value through direct alteration of the accrued data set and to provide data that a policy-learning neural network is able to fit appropriately. The proposed algorithm was validated with real robot scenarios while learning from inconsistent demonstrations, where the resulting policies consistently achieved their prescribed objectives. A video showing our method and experiments can be found at: https://youtu.be/oGYnzlW9Ncw.

Details

ICRA Conference 2020 Conference Paper

Predicting and Optimizing Ergonomics in Physical Human-Robot Cooperation Tasks

  • Linda F. van der Spaa
  • Michael Gienger
  • Tamas Bates
  • Jens Kober

This paper presents a method to incorporate ergonomics into the optimization of action sequences for bi-manual human-robot cooperation tasks with continuous physical interaction. Our first contribution is a novel computational model of the human that allows prediction of an ergonomics assessment corresponding to each step in a task. The model is learned from human motion capture data in order to predict the human pose as realistically as possible. The second contribution is a combination of this prediction model with an informed graph search algorithm, which allows computation of human-robot cooperative plans with improved ergonomics according to the incorporated method for ergonomic assessment. The concepts have been evaluated in simulation and in a small user study in which the subjects manipulate a large object with a 32 DoF bimanual mobile robot as partner. For all subjects, the ergonomic-enhanced planner shows their reduced ergonomic cost compared to a baseline planner.

Details

ICRA Conference 2019 Conference Paper

Continuous Control for High-Dimensional State Spaces: An Interactive Learning Approach

  • Rodrigo Pérez-Dattari
  • Carlos Celemin
  • Javier Ruiz-del-Solar
  • Jens Kober

Deep Reinforcement Learning (DRL) has become a powerful methodology to solve complex decision-making problems. However, DRL has several limitations when used in real-world problems (e. g. , robotics applications). For instance, long training times are required and cannot be accelerated in contrast to simulated environments, and reward functions may be hard to specify/model and/or to compute. Moreover, the transfer of policies learned in a simulator to the real-world has limitations (reality gap). On the other hand, machine learning methods that rely on the transfer of human knowledge to an agent have shown to be time efficient for obtaining well performing policies and do not require a reward function. In this context, we analyze the use of human corrective feedback during task execution to learn policies with high-dimensional state spaces, by using the D-COACH framework, and we propose new variants of this framework. D-COACH is a Deep Learning based extension of COACH (COrrective Advice Communicated by Humans), where humans are able to shape policies through corrective advice. The enhanced version of DCOACH, which is proposed in this paper, largely reduces the time and effort of a human for training a policy. Experimental results validate the efficiency of the D-COACH framework in three different problems (simulated and with real robots), and show that its enhanced version reduces the human training effort considerably, and makes it feasible to learn policies within periods of time in which a DRL agent do not reach any improvement.

Details

EAAI Journal 2019 Journal Article

Reinforcement learning based compensation methods for robot manipulators

  • Yudha P. Pane
  • Subramanya P. Nageshrao
  • Jens Kober
  • Robert Babuška

Smart robotics will be a core feature while migrating from Industry 3. 0 (i. e. , mass manufacturing) to Industry 4. 0 (i. e. , customized or social manufacturing). A key characteristic of a smart system is its ability to learn. For smart manufacturing, this means incorporating learning capabilities into the current fixed, repetitive, task-oriented industrial manipulators, thus rendering them ‘smart’. In this paper we introduce two reinforcement learning (RL) based compensation methods. The learned correction signal, which compensates for unmodeled aberrations, is added to the existing nominal input with an objective to enhance the control performance. The proposed learning algorithms are evaluated on a 6-DoF industrial robotic manipulator arm to follow different kinds of reference paths, such as square or a circular path, or to track a trajectory on a three dimensional surface. In an extensive experimental study we compare the performance of our learning-based methods with well-known tracking controllers, namely, proportional-derivative (PD), model predictive control (MPC), and iterative learning control (ILC). The experimental results show a considerable performance improvement thanks to our RL-based methods when compared to PD, MPC, and ILC.

Details DOI

JMLR Journal 2018 Journal Article

Experience Selection in Deep Reinforcement Learning for Control

  • Tim de Bruin
  • Jens Kober
  • Karl Tuyls
  • Robert Babuška

Experience replay is a technique that allows off-policy reinforcement-learning methods to reuse past experiences. The stability and speed of convergence of reinforcement learning, as well as the eventual performance of the learned policy, are strongly dependent on the experiences being replayed. Which experiences are replayed depends on two important choices. The first is which and how many experiences to retain in the experience replay buffer. The second choice is how to sample the experiences that are to be replayed from that buffer. We propose new methods for the combined problem of experience retention and experience sampling. We refer to the combination as experience selection. We focus our investigation specifically on the control of physical systems, such as robots, where exploration is costly. To determine which experiences to keep and which to replay, we investigate different proxies for their immediate and long-term utility. These proxies include age, temporal difference error and the strength of the applied exploration noise. Since no currently available method works in all situations, we propose guidelines for using prior knowledge about the characteristics of the control problem at hand to choose the appropriate experience replay strategy. [abs] [ pdf ][ bib ] &copy JMLR 2018. ( edit, beta )

PDF Details

IROS Conference 2018 Conference Paper

Human-Robot Cooperative Object Manipulation with Contact Changes

  • Michael Gienger
  • Dirk Ruiken
  • Tamas Bates
  • Mohamed Regaieg
  • Michael MeiBner
  • Jens Kober
  • Philipp Seiwald
  • Arne-Christoph Hildebrandt

This paper presents a system for cooperatively manipulating large objects between a human and a robot. This physical interaction system is designed to handle, transport, or manipulate large objects of different shapes in cooperation with a human. Unique points are the bi-manual physical cooperation, the sequential characteristic of the cooperation including contact changes, and a novel architecture combining force interaction cues, interactive search-based planning, and online trajectory and motion generation. The resulting system implements a mixed initiative collaboration strategy, deferring to the human when his intentions are unclear, and driving the task once understood. This results in an easy and intuitive human-robot interaction. It is evaluated in simulations and on a bi-manual mobile robot with 32 degrees of freedom.

Details

IROS Conference 2016 Conference Paper

Improved deep reinforcement learning for robotics through distribution-based experience retention

  • Tim de Bruin
  • Jens Kober
  • Karl Tuyls
  • Robert Babuska

Recent years have seen a growing interest in the use of deep neural networks as function approximators in reinforcement learning. In this paper, an experience replay method is proposed that ensures that the distribution of the experiences used for training is between that of the policy and a uniform distribution. Through experiments on a magnetic manipulation task it is shown that the method reduces the need for sustained exhaustive exploration during learning. This makes it attractive in scenarios where sustained exploration is in-feasible or undesirable, such as for physical systems like robots and for life long learning. The method is also shown to improve the generalization performance of the trained policy, which can make it attractive for transfer learning. Finally, for small experience databases the method performs favorably when compared to the recently proposed alternative of using the temporal difference error to determine the experience sample distribution, which makes it an attractive option for robots with limited memory capacity.

Details

IROS Conference 2016 Conference Paper

Probabilistic decomposition of sequential force interaction tasks into Movement Primitives

  • Simon Manschitz
  • Michael Gienger
  • Jens Kober
  • Jan Peters 0001

Learning sequential force interaction tasks from kinesthetic demonstrations is a promising approach to transfer human manipulation abilities to a robot. In this paper we propose a novel concept to decompose such demonstrations into a set of Movement Primitives (MPs). The decomposition is based on a probability distribution we call Directional Normal Distribution (DND). To capture the sequential properties of the manipulation task, we model the demonstrations with a Hidden Markov Model (HMM). Here, we employ mixtures of DNDs as the HMM's output emissions. The combination of HMMs and mixtures of DNDs allows to infer the MP's composition, i. e. , its coordinate frames, control variables and target coordinates from the demonstration data. In addition, it permits to determine an appropriate number of MPs that explains the demonstrations best. We evaluate the approach on kinesthetic demonstrations of a light bulb unscrewing task. Decomposing the task leads to intuitive and meaningful MPs that reflect the natural structure of the task.

Details

ICRA Conference 2015 Conference Paper

Learning movement primitives for force interaction tasks

  • Jens Kober
  • Michael Gienger
  • Jochen J. Steil

Kinesthetic teaching is a promising approach to acquire robot skills in an intuitive way. This paper focuses on learning skills that do not solely rely on kinematics but also need to take into account interaction forces. We present three novel concepts towards learning such force interaction skills. Firstly, we determine segments from a small number of continuous kinesthetic demonstrations using contact information. Secondly, we associate each segment with a movement primitive, and determine its composition, i. e. , the control variables and reference frames that allow to reproduce the demonstrated task. Lastly, we propose a concept to determine the transitions between the primitives during reproduction. The proposed methods are evaluated on a box pulling and flipping task, and show very good generalization abilities for objects with different geometries, and situations with different object arrangements.

Details

IROS Conference 2015 Conference Paper

Probabilistic progress prediction and sequencing of concurrent movement primitives

  • Simon Manschitz
  • Jens Kober
  • Michael Gienger
  • Jan Peters 0001

Classical approaches towards learning coordinated movement tasks often represent a movement in a sequential and exclusive fashion. Introducing concurrency allows to decompose such tasks into a number of separate sequences, for instance for two different end-effectors. While this results in a compact and generic representation of the individual movement primitives (MPs), it is a hard problem to learn their temporal and causal organization. This paper presents a concept for learning movement tasks that require the coordination of several controlled effectors of a robot. We firstly introduce a concept to learn and estimate the progress of individual MPs from a low number of demonstrations. Secondly, we propose a representation of the task that incorporates several concurrent sequences of MPs. Combining these two elements allows to learn and reproduce coordinated bi-manual movement tasks robustly. The synchronization of the concurrent MPs is achieved implicitly using the progress prediction. The approach is evaluated in two simulation studies with a 25 degrees of freedom two-arm robot performing a pick-and-place task.

Details

IROS Conference 2014 Conference Paper

Learning to sequence movement primitives from demonstrations

  • Simon Manschitz
  • Jens Kober
  • Michael Gienger
  • Jan Peters 0001

We present an approach for learning sequential robot skills through kinesthetic teaching. The demonstrations are represented by a sequence graph. Finding the transitions between consecutive basic movements is treated as classification problem where both Support Vector Machines and Gaussian Mixture Models are evaluated as classifiers. We show how the observed primitive order of all demonstrations can help to improve the movement reproduction by restricting the classification outcome to the currently executed primitive and its possible successors in the graph. The approach is validated with an experiment in which a 7-DOF Barrett WAM robot learns to unscrew a light bulb.

Details

IROS Conference 2012 Conference Paper

Learning throwing and catching skills

  • Jens Kober
  • Katharina Muelling
  • Jan Peters 0001

In this video, we present approaches for learning throwing and catching skills. We first show how a hitting skill (i. e. , table tennis) can be learned using a combination of imitation and reinforcement learning. This hitting skill is subsequently generalized to a catching skill. Secondly, we show how a robot can adapt a throwing skill to new targets. Finally, we demonstrate that a BioRob and a Barrett WAM can play catch together using the previously acquired skills.

Details

ECAI Conference 2012 Conference Paper

Robot Skill Learning

  • Jan Peters 0001
  • Katharina Muelling
  • Jens Kober
  • Duy Nguyen-Tuong
  • Oliver Kroemer

Learning robots that can acquire new motor skills and refine existing ones have been a long standing vision of robotics, artificial intelligence, and the cognitive sciences. Early steps towards this goal in the 1980s made clear that reasoning and human insights will not suffice. Instead, new hope has been offered by the rise of modern machine learning approaches. However, to date, it becomes increasingly clear that off-the-shelf machine learning approaches will not be adequate for robot skill learning as these methods often do not scale into the high-dimensional domains of manipulator and humanoid robotics, nor do they fulfill the real-time requirement of the domain. As an alternative, we propose to divide the generic skill learning problem into parts that can be well-understood from a robotics point of view. After designing appropriate learning approaches for these basic components, these will serve as the ingredients of a general approach to robot skill learning. In this paper, we discuss our recent and current progress in this direction. As such, we present our work on learning to control, learning elementary movements, as well as our steps towards the learning of complex tasks. We show several evaluations using both real robots as well as physically realistic simulations.

Details

IROS Conference 2011 Conference Paper

Learning elementary movements jointly with a higher level task

  • Jens Kober
  • Jan Peters 0001

Many motor skills consist of many lower level elementary movements that need to be sequenced in order to achieve a task. In order to learn such a task, both the primitive movements as well as the higher-level strategy need to be acquired at the same time. In contrast, most learning approaches focus either on learning to combine a fixed set of options or to learn just single options. In this paper, we discuss a new approach that allows improving the performance of lower level actions while pursuing a higher level task. The presented approach is applicable to learning a wider range motor skills, but in this paper, we employ it for learning games where the player wants to improve his performance at the individual actions of the game while still performing well at the strategy level game. We propose to learn the lower level actions using Cost-regularized Kernel Regression and the higher level actions using a form of Policy Iteration. The two approaches are coupled by their transition probabilities. We evaluate the approach on a side-stall-style throwing game both in simulation and with a real BioRob.

Details

IJCAI Conference 2011 Conference Paper

Reinforcement Learning to Adjust Robot Movements to New Situations

  • Jens Kober
  • Erhan Oztop
  • Jan Peters

Many complex robot motor skills can be represented using elementary movements, and there exist efficient techniques for learning parametrized motor plans using demonstrations and self-improvement. However with current techniques, in many cases, the robot currently needs to learn a new elementary movement even if a parametrized motor plan exists that covers a related situation. A method is needed that modulates the elementary movement through the meta-parameters of its representation. In this paper, we describe how to learn such mappings from circumstances to meta-parameters using reinforcement learning. In particular we use a kernelized version of the reward-weighted regression. We show two robot applications of the presented setup in robotic domains; the generalization of throwing movements in darts, and of hitting movements in table tennis. We demonstrate that both tasks can be learned successfully using simulated and real robots.

PDF Details DOI

IROS Conference 2010 Conference Paper

A biomimetic approach to robot table tennis

  • Katharina Muelling
  • Jens Kober
  • Jan Peters 0001

Although human beings see and move slower than table tennis or baseball robots, they manage to outperform such robot systems. One important aspect of this better performance is the human movement generation. In this paper, we study trajectory generation for table tennis from a biomimetic point of view. Our focus lies on generating efficient stroke movements capable of mastering variations in the environmental conditions, such as changing ball speed, spin and position. We study table tennis from a human motor control point of view. To make headway towards this goal, we construct a trajectory generator for a single stroke using the discrete movement stages hypothesis and the virtual hitting point hypothesis to create a model that produces a human-like stroke movement. We verify the functionality of the trajectory generator for a single forehand stroke both in a simulation and using a real Barrett WAM™.

Details

ICRA Conference 2010 Conference Paper

Movement templates for learning of hitting and batting

  • Jens Kober
  • Katharina Muelling
  • Oliver Kroemer
  • Christoph H. Lampert
  • Bernhard Schölkopf
  • Jan Peters 0001

Hitting and batting tasks, such as tennis forehands, ping-pong strokes, or baseball batting, depend on predictions where the ball can be intercepted and how it can properly be returned to the opponent. These predictions get more accurate over time, hence the behaviors need to be continuously modified. As a result, movement templates with a learned global shape need to be adapted during the execution so that the racket reaches a target position and velocity that will return the ball over to the other side of the net or court. It requires altering learned movements to hit a varying target with the necessary velocity at a specific instant in time. Such a task cannot be incorporated straightforwardly in most movement representations suitable for learning. For example, the standard formulation of the dynamical system based motor primitives (introduced by Ijspeert et al. [1]) does not satisfy this property despite their flexibility which has allowed learning tasks ranging from locomotion to kendama. In order to fulfill this requirement, we reformulate the Ijspeert framework to incorporate the possibility of specifying a desired hitting point and a desired hitting velocity while maintaining all advantages of the original formulation. We show that the proposed movement template formulation works well in two scenarios, i. e. , for hitting a ball on a string with a table tennis racket at a specified velocity and for returning balls launched by a ball gun successfully over the net using forehand movements. All experiments were carried out on a Barrett WAM using a four camera vision system.

Details

ICRA Conference 2009 Conference Paper

Learning motor primitives for robotics

  • Jens Kober
  • Jan Peters 0001

The acquisition and self-improvement of novel motor skills is among the most important problems in robotics. Motor primitives offer one of the most promising frameworks for the application of machine learning techniques in this context. Employing an improved form of the dynamic systems motor primitives originally introduced by Ijspeert et al. [2], we show how both discrete and rhythmic tasks can be learned using a concerted approach of both imitation and reinforcement learning. For doing so, we present both learning algorithms and representations targeted for the practical application in robotics. Furthermore, we show that it is possible to include a start-up phase in rhythmic primitives. We show that two new motor skills, i. e. , Ball-in-a-Cup and Ball-Paddling, can be learned on a real Barrett WAM robot arm at a pace similar to human learning while achieving a significantly more reliable final performance.

Details

IROS Conference 2008 Conference Paper

Learning perceptual coupling for motor primitives

  • Jens Kober
  • Betty J. Mohler
  • Jan Peters 0001

Dynamic system-based motor primitives [1] have enabled robots to learn complex tasks ranging from Tennis-swings to locomotion. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus, and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. In this paper, we propose an augmented version of the dynamic systems motor primitives which incorporates perceptual coupling to an external variable. The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using a novel reinforcement learning method which is particularly well-suited for motor primitives.

Details

EWRL Workshop 2008 Conference Paper

Policy Learning - A Unified Perspective with Applications in Robotics

  • Jan Peters 0001
  • Jens Kober
  • Duy Nguyen-Tuong

Abstract Policy Learning approaches are among the best suited methods for high-dimensional, continuous control systems such as anthropomorphic robot arms and humanoid robots. In this paper, we show two contributions: firstly, we show a unified perspective which allows us to derive several policy learning algorithms from a common point of view, i. e, policy gradient algorithms, natural-gradient algorithms and EM-like policy learning. Secondly, we present several applications to both robot motor primitive learning as well as to robot control in task space. Results both from simulation and several different real robots are shown.

Details

NeurIPS Conference 2008 Conference Paper

Policy Search for Motor Primitives in Robotics

  • Jens Kober
  • Jan Peters

Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results into a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning by assuming a form of exploration that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable in complex motor learning tasks. We compare this algorithm to alternative parametrized policy search methods and show that it outperforms previous methods. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM robot arm.

PDF Details

NeurIPS Conference 2008 Conference Paper

Using Bayesian Dynamical Systems for Motion Template Libraries

  • Silvia Chiappa
  • Jens Kober
  • Jan Peters

Motor primitives or motion templates have become an important concept for both modeling human motor control as well as generating robot behaviors using imitation learning. Recent impressive results range from humanoid robot movement generation to timing models of human motions. The automatic generation of skill libraries containing multiple motion templates is an important step in robot learning. Such a skill learning system needs to cluster similar movements together and represent each resulting motion template as a generative model which is subsequently used for the execution of the behavior by a robot system. In this paper, we show how human trajectories captured as multidimensional time-series can be clustered using Bayesian mixtures of linear Gaussian state-space models based on the similarity of their dynamics. The appropriate number of templates is automatically determined by enforcing a parsimonious parametrization. As the resulting model is intractable, we introduce a novel approximation method based on variational Bayes, which is especially designed to enable the use of efficient inference algorithms. On recorded human Balero movements, this method is not only capable of finding reasonable motion templates but also yields a generative model which works well in the execution of this complex task on a simulated anthropomorphic SARCOS arm.

PDF Details
v2026.09.13