Marc Toussaint

TMLR Journal 2025 Journal Article

Global Safe Sequential Learning via Efficient Knowledge Transfer

• Cen-You Li
• Olaf Dünnbier
• Marc Toussaint
• Barbara Rakitsch
• Christoph Zimmer

Sequential learning methods, such as active learning and Bayesian optimization, aim to select the most informative data for task learning. In many applications, however, data selection is constrained by unknown safety conditions, motivating the development of safe learning approaches. A promising line of safe learning methods uses Gaussian processes to model safety conditions, restricting data selection to areas with high safety confidence. However, these methods are limited to local exploration around an initial seed dataset, as safety confidence centers around observed data points. As a consequence, task exploration is slowed down and safe regions disconnected from the initial seed dataset remain unexplored. In this paper, we propose safe transfer sequential learning to accelerate task learning and to expand the explorable safe region. By leveraging abundant offline data from a related source task, our approach guides exploration in the target task more effectively. We also provide a theoretical analysis to explain why single-task method cannot cope with disconnected regions. Finally, we introduce a computationally efficient approximation of our method that reduces runtime through pre-computations. Our experiments demonstrate that this approach, compared to state-of-the-art methods, learns tasks with lower data consumption and enhances global exploration across multiple disjoint safe regions, while maintaining comparable computational efficiency.

IROS Conference 2025 Conference Paper

Regrasp Maps for Sequential Manipulation Planning

• Svetlana Levit
• Marc Toussaint

We consider manipulation problems in constrained and cluttered settings, which require several regrasps at unknown locations. We propose to inform an optimization-based task and motion planning (TAMP) solver with possible regrasp areas and grasp sequences to speed up the search. Our main idea is to use a state space abstraction, a regrasp map, capturing the combinations of available grasps in different parts of the configuration space, and allowing us to provide the solver with guesses for the mode switches and additional constraints for the object placements. By interleaving the creation of regrasp maps, their adaptation based on failed refinements, and solving TAMP (sub)problems, we are able to provide a robust search method for challenging regrasp manipulation problems.

UAI Conference 2025 Conference Paper

Stein Variational Evolution Strategies

• Cornelius V. Braun
• Robert Tjarko Lange
• Marc Toussaint

Efficient global optimization and sampling are fundamental challenges, particularly in fields such as robotics and reinforcement learning, where gradients may be unavailable or unreliable. In this context, jointly optimizing multiple solutions is a promising approach to avoid local optima. While Stein Variational Gradient Descent (SVGD) provides a powerful framework for sampling diverse solutions, its reliance on first-order information limits its applicability to differentiable objectives. Existing gradient-free SVGD variants often suffer from slow convergence, and poor scalability. To improve gradient-free sampling and optimization, we propose Stein Variational CMA-ES, a novel gradient-free SVGD-like method that combines the efficiency of evolution strategies with SVGD-based repulsion forces. We perform an extensive empirical evaluation across several domains, which shows that the integration of the ES update in SVGD significantly improves the performance on multiple challenging benchmark problems. Our findings establish SV-CMA-ES as a scalable method for zero-order sampling and blackbox optimization, bridging the gap between SVGD and evolution strategies.

IROS Conference 2024 Conference Paper

Camera-Based Belief Space Planning in Discrete Partially-Observable Domains

• Janis Eric Freund
• Camille Phiquepal
• Andreas Orthey
• Marc Toussaint

Robots often have to operate in discrete partially observable worlds, where the state of the world is only observable at runtime. To react to different world states, robots need contingencies. To find contingencies, prior work developed the path tree optimization (PTO) method, which computes motion contingencies by constructing a tree of motion paths in belief space. In this paper, we extend upon PTO by enabling camera-based belief space planning through an extension of the open motion planning library (OMPL). By leveraging this extension, we develop an improved camera-based state sampler and an efficient open-source implementation of PTO. This version of PTO supports a virtual camera, non-euclidean state spaces, and different state samplers. We evaluate this improved version of PTO on four realistic scenarios with a virtual camera in up to 10-dimensional state spaces. In our evaluations, we compare PTO both with a default and with the new camera-based state sampler. The results indicate that the camera-based state sampler improves success rates in 3 out of 4 scenarios while having a significant lower memory footprint. Our work thus makes an important step in advancing belief-space planning and provides researchers with an open source tool to use, modify, and benchmark belief-space planning methods.

ICRA Conference 2024 Conference Paper

db-CBS: Discontinuity-Bounded Conflict-Based Search for Multi-Robot Kinodynamic Motion Planning

• Akmaral Moldagalieva
• Joaquim Ortiz de Haro
• Marc Toussaint
• Wolfgang Hönig

This paper presents a multi-robot kinodynamic motion planner that enables a team of robots with different dynamics, actuation limits, and shapes to reach their goals in challenging environments. We solve this problem by combining Conflict-Based Search (CBS), a multi-agent path finding method, and discontinuity-bounded A*, a single-robot kinodynamic motion planner. Our method, db-CBS, operates in three levels. Initially, we compute trajectories for individual robots using a graph search that allows bounded discontinuities between precomputed motion primitives. The second level identifies inter-robot collisions and resolves them by imposing constraints on the first level. The third and final level uses the resulting solution with discontinuities as an initial guess for a joint space trajectory optimization. The procedure is repeated with a reduced discontinuity bound. Our approach is anytime, probabilistically complete, asymptotically optimal, and finds near-optimal solutions quickly. Experimental results with robot dynamics such as unicycle, double integrator, and car with trailer in different settings show that our method is capable of solving challenging tasks with a higher success rate and lower cost than the existing state-of-the-art.

ICRA Conference 2024 Conference Paper

Effort Level Search in Infinite Completion Trees with Application to Task-and-Motion Planning

• Marc Toussaint
• Joaquim Ortiz de Haro
• Valentin N. Hartmann
• Erez Karpas
• Wolfgang Hönig

Solving a Task-and-Motion Planning (TAMP) problem can be represented as a sequential (meta-) decision process, where early decisions concern the skeleton (sequence of logic actions) and later decisions concern what to compute for such skeletons (e. g. , action parameters, bounds, RRT paths, or full optimal manipulation trajectories). We consider the general problem of how to schedule compute effort in such hierarchical solution processes. More specifically, we introduce infinite completion trees as a problem formalization, where before we can expand or evaluate a node, we have to solve a preemptible computational sub-problem of a priori unknown compute effort. Infinite branchings represent an infinite choice of random initializations of computational sub-problems. Decision making in such trees means to decide on where to invest compute or where to widen a branch. We propose a heuristic to balance branching width and compute depth using polynomial level sets. We show completeness of the resulting solver and that a round robin baseline strategy used previously for TAMP becomes a special case. Experiments confirm the robustness and efficiency of the method on problems including stochastic bandits and a suite of TAMP problems, and compare our approach to a round robin baseline. An appendix comparing the framework to bandit methods and proposing a corresponding tree policy version is found on the supplementary webpage 1.

IROS Conference 2024 Conference Paper

GSRM: Building Roadmaps for Query-Efficient and Near-Optimal Path Planning Using a Reaction Diffusion System

• Christian Henkel
• Marc Toussaint
• Wolfgang Hönig

Mobile robots frequently navigate on roadmaps, i. e. , graphs where edges represent safe motions, in applications such as healthcare, hospitality, and warehouse automation. Often the environment is quasi-static, i. e. , it is sufficient to construct a roadmap once and then use it for any future planning queries. Roadmaps are typically used with graph search algorithm to find feasible paths for the robots. Therefore, the roadmap should be well-connected, and graph searches should produce near-optimal solutions with short solution paths while simultaneously be computationally efficient to execute queries quickly. We propose a new method to construct roadmaps based on the Gray-Scott reaction diffusion system and Delaunay triangulation. Our approach, GSRM, produces roadmaps with evenly distributed vertices and edges that are well-connected even in environments with challenging narrow passages. Empirically, we compare to classical roadmaps generated by 8-connected grids, probabilistic roadmaps (PRM, SPARS2), and optimized roadmap graphs (ORM). Our results show that GSRM consistently produces superior roadmaps that are well-connected, have high query efficiency, and result in short solution paths.

IROS Conference 2024 Conference Paper

iDb-RRT: Sampling-based Kinodynamic Motion Planning with Motion Primitives and Trajectory Optimization

• Joaquim Ortiz de Haro
• Wolfgang Hönig
• Valentin N. Hartmann
• Marc Toussaint
• Ludovic Righetti

Rapidly-exploring Random Trees (RRT) and its variations have emerged as a robust and efficient tool for finding collision-free paths in robotic systems. However, adding dynamic constraints makes the motion planning problem significantly harder, as it requires solving two-value boundary problems (computationally expensive) or propagating random control inputs (uninformative). Alternatively, Iterative Discontinuity Bounded A* (iDb-A*), introduced in our previous study, combines search and optimization iteratively. The search step connects short trajectories (motion primitives) while allowing a bounded discontinuity between the motion primitives, which is later repaired in the trajectory optimization step. Building upon these foundations, in this paper, we present iDb-RRT, a sampling-based kinodynamic motion planning algorithm that combines motion primitives and trajectory optimization within the RRT framework. iDb-RRT is probabilistically complete and can be implemented in forward or bidirectional mode. We have tested our algorithm across a benchmark suite comprising 30 problems, spanning 8 different systems, and shown that iDb-RRT can find solutions up to 10x faster than previous methods, especially in complex scenarios that require long trajectories or involve navigating through narrow passages.

IROS Conference 2024 Conference Paper

Kinodynamic Motion Planning for a Team of Multirotors Transporting a Cable-Suspended Payload in Cluttered Environments

• Khaled Wahba
• Joaquim Ortiz de Haro
• Marc Toussaint
• Wolfgang Hönig

We propose a motion planner for cable-driven payload transportation using multiple unmanned aerial vehicles (UAVs) in an environment cluttered with obstacles. Our planner is kinodynamic, i. e. , it considers the full dynamics model of the transporting system including actuation constraints. Due to the high dimensionality of the planning problem, we use a hierarchical approach where we first solve for the geometric motion using a sampling-based method with a novel sampler, followed by constrained trajectory optimization that considers the full dynamics of the system. Both planning stages consider inter-robot and robot/obstacle collisions. We demonstrate in a software-in-the-loop simulation and real flight experiments that there is a significant benefit in kinodynamic motion planning for such payload transport systems with respect to payload tracking error and energy consumption compared to the standard methods of planning for the payload alone. Notably, we observe a significantly higher success rate in scenarios where the team formation changes are needed to move through tight spaces.

ICRA Conference 2024 Conference Paper

Solving Sequential Manipulation Puzzles by Finding Easier Subproblems

• Svetlana Levit
• Joaquim Ortiz de Haro
• Marc Toussaint

We consider a set of challenging sequential manipulation puzzles, where an agent has to interact with multiple movable objects and navigate narrow passages. Such settings are notoriously difficult for Task-and-Motion Planners, as they require interdependent regrasps and solving hard motion planning problems. In this paper, we propose to search over sequences of easier pick-and-place subproblems, which can lead to the solution of the manipulation puzzle. Our method combines a heuristic-driven forward search of subproblems with an optimization-based Task-and-Motion Planning solver. To guide the search, we introduce heuristics to generate and prioritize useful subgoals. We evaluate our approach on various manually designed and automatically generated scenes, demonstrating the benefits of auxiliary subproblems in sequential manipulation planning.

IROS Conference 2023 Conference Paper

Efficient Path Planning In Manipulation Planning Problems by Actively Reusing Validation Effort

• Valentin N. Hartmann
• Joaquim Ortiz de Haro
• Marc Toussaint

The path planning problems arising in manipulation planning and in task and motion planning settings are typically repetitive: the same manipulator moves in a space that only changes slightly. Despite this potential for reuse of information, few planners fully exploit the available information. To better enable this reuse, we decompose the collision checking into reusable, and non-reusable parts. We then treat the sequences of path planning problems in manipulation planning as a multiquery path planning problem. This allows the usage of planners that actively minimize planning effort over multiple queries, and by doing so, actively reuse previous knowledge. We implement this approach in EIRM* and effort ordered LazyPRM*, and benchmark it on multiple simulated robotic examples. Further, we show that the approach of decomposing collision checks additionally enables the reuse of the gained knowledge over multiple different instances of the same problem, i. e. , in a multiquery manipulation planning scenario. The planners using the decomposed collision checking out-perform the other planners in initial solution time by up to a factor of two while providing a similar solution quality.

ICRA Conference 2023 Conference Paper

Learning Feasibility of Factored Nonlinear Programs in Robotic Manipulation Planning

• Joaquim Ortiz de Haro
• Jung-Su Ha
• Danny Driess
• Erez Karpas
• Marc Toussaint

A factored Nonlinear Program (Factored-NLP) explicitly models the dependencies between a set of continuous variables and nonlinear constraints, providing an expressive formulation for relevant robotics problems such as manipulation planning or simultaneous localization and mapping. When the problem is over-constrained or infeasible, a fundamental issue is to detect a minimal subset of variables and constraints that are infeasible. Previous approaches require solving several nonlinear programs, incrementally adding and removing constraints, and are thus computationally expensive. In this paper, we propose a graph neural architecture that predicts which variables and constraints are jointly infeasible. The model is trained with a dataset of labeled subgraphs of Factored-NLPs, and importantly, can make useful predictions on larger factored nonlinear programs than the ones seen during training. We evaluate our approach in robotic manipulation planning, where our model is able to generalize to longer manipulation sequences involving more objects and robots, and different geometric environments. The experiments show that the learned model accelerates general algorithms for conflict extraction (by a factor of 50) and heuristic algorithms that exploit expert knowledge (by a factor of 4).

ICML Conference 2023 Conference Paper

PaLM-E: An Embodied Multimodal Language Model

• Danny Driess
• Fei Xia 0002
• Mehdi S. M. Sajjadi
• Corey Lynch
• Aakanksha Chowdhery
• Brian Ichter
• Ayzaan Wahid
• Jonathan Tompson

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e. g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multimodal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

IROS Conference 2023 Conference Paper

Spatial Reasoning via Deep Vision Models for Robotic Sequential Manipulation

• Hongyou Zhou
• Ingmar Schubert
• Marc Toussaint
• Ozgur S. Oguz

In this paper, we propose using deep neural architectures (i. e. , vision transformers and ResNet) as heuristics for sequential decision-making in robotic manipulation problems. This formulation enables predicting the subset of objects that are relevant for completing a task. Such problems are often addressed by task and motion planning (TAMP) formulations combining symbolic reasoning and continuous motion planning. In essence, the action-object relationships are resolved for discrete, symbolic decisions that are used to solve manipulation motions (e. g. , via nonlinear trajectory optimization). However, solving long-horizon tasks requires consideration of all possible action-object combinations which limits the scalability of TAMP approaches. To overcome this combinatorial complexity, we introduce a visual perception module integrated with a TAMP-solver. Given a task and an initial image of the scene, the learned model outputs the relevancy of objects to accomplish the task. By incorporating the predictions of the model into a TAMP formulation as a heuristic, the size of the search space is significantly reduced. Results show that our framework finds feasible solutions more efficiently when compared to a state-of-the-art TAMP solver.

IROS Conference 2022 Conference Paper

BITKOMO: Combining Sampling and Optimization for Fast Convergence in Optimal Motion Planning

• Jay Kamat
• Joaquim Ortiz de Haro
• Marc Toussaint
• Florian T. Pokorny
• Andreas Orthey

Optimal sampling based motion planning and trajectory optimization are two competing frameworks to generate optimal motion plans. Both frameworks have complementary properties: Sampling based planners are typically slow to converge, but provide optimality guarantees. Trajectory optimizers, however, are typically fast to converge, but do not provide global optimality guarantees in nonconvex problems, e. g. scenarios with obstacles. To achieve the best of both worlds, we introduce a new planner, BITKOMO, which integrates the asymptotically optimal Batch Informed Trees (BIT*) planner with the K-Order Markov Optimization (KOMO) trajectory optimization framework. Our planner is anytime and maintains the same asymptotic optimality guarantees provided by BIT*, while also exploiting the fast convergence of the KOMO trajectory optimizer. We experimentally evaluate our planner on manipulation scenarios that involve high dimensional configuration spaces, with up to two 7-DoF manipulators, obstacles and narrow passages. BITKOMO performs better than KOMO by succeeding even when KOMO fails, and it outperforms BIT* in terms of convergence to the optimal solution.

ICAPS Conference 2022 Conference Paper

Conflict-Directed Diverse Planning for Logic-Geometric Programming

• Joaquim Ortiz de Haro
• Erez Karpas
• Marc Toussaint
• Michael Katz 0001

Robots operating in the real world must combine task planning for reasoning about what to do with motion planning for reasoning about how to do it -- this is known as task and motion planning. One promising approach for task and motion planning is Logic Geometric Programming (LGP) which integrates a logical layer and a geometric layer in an optimization formulation. The logical layer describes feasible high-level actions at an abstract symbolic level, while the geometric layer uses continuous optimization methods to reason about motion trajectories with geometric constraints. In this paper we propose a new approach for solving task and motion planning problems in the LGP formulation, that leverages state-of-the-art diverse planning at the logical layer to explore the space of feasible logical plans, and minimizes the number of optimization problems to be solved on the continuous geometric layer. To this end, geometric infeasibility is fed back into planning by identifying prefix conflicts and incorporating this back into the planner through a novel multi-prefix forbidding compilation. We further leverage diverse planning with a new novelty criteria for selecting candidate plans based on the prefix novelty, and a metareasoning approach which attempts to extract only useful conflicts by leveraging the information that is gathered in the course of solving the given problem.

IROS Conference 2022 Conference Paper

db-A*: Discontinuity-bounded Search for Kinodynamic Mobile Robot Motion Planning

• Wolfgang Hönig
• Joaquim Ortiz de Haro
• Marc Toussaint

We consider time-optimal motion planning for dynamical systems that are translation-invariant, a property that holds for many mobile robots, such as differential-drives, cars, airplanes, and multirotors. Our key insight is that we can extend graph-search algorithms to the continuous case when used symbiotically with optimization. For the graph search, we introduce discontinuity-bounded A* (db-A*), a generalization of the A* algorithm that uses concepts and data structures from sampling-based planners. Db-A* reuses short trajectories, so-called motion primitives, as edges and allows a maximum user-specified discontinuity at the vertices. These trajectories are locally repaired with trajectory optimization, which also provides new improved motion primitives. Our novel kinodynamic motion planner, kMP-db-A*, has almost surely asymptotic optimal behavior and computes near-optimal solutions quickly. For our empirical validation, we provide the first benchmark that compares search-, sampling-, and optimization- based time-optimal motion planning on multiple dynamical systems in different settings. Compared to the baselines, kMP- db-A* consistently solves more problem instances, finds lower- cost initial solutions, and converges more quickly.

IROS Conference 2022 Conference Paper

FC 3: Feasibility-Based Control Chain Coordination

• Jason Harris
• Danny Driess
• Marc Toussaint

Hierarchical coordination of controllers often uses symbolic state representations that fully abstract their underlying low-level controllers, treating them as “black boxes” to the symbolic action abstraction. This paper proposes a framework to realize robust behavior, which we call Feasibility-based Control Chain Coordination (FC 3 ). Our controllers expose the geometric features and constraints they operate on. Based on this, FC 3 can reason over the controllers' feasibility and their sequence feasibility. For a given task, FC 3 first automatically constructs a library of potential controller chains using a symbolic action tree, which is then used to coordinate controllers in a chain, evaluate task feasibility, as well as switching between controller chains if necessary. In several real-world experiments we demonstrate FC 3, s robustness and awareness of the task's feasibility through its own actions and gradual responses to different interferences.

NeurIPS Conference 2022 Conference Paper

Reinforcement Learning with Neural Radiance Fields

• Danny Driess
• Ingmar Schubert
• Pete Florence
• Yunzhu Li
• Marc Toussaint

It is a long-standing problem to find effective representations for training reinforcement learning (RL) agents. This paper demonstrates that learning state representations with supervision from Neural Radiance Fields (NeRFs) can improve the performance of RL compared to other learned representations or even low-dimensional, hand-engineered state information. Specifically, we propose to train an encoder that maps multiple image observations to a latent space describing the objects in the scene. The decoder built from a latent-conditioned NeRF serves as the supervision signal to learn the latent space. An RL algorithm then operates on the learned latent space as its state representation. We call this NeRF-RL. Our experiments indicate that NeRF as supervision leads to a latent space better suited for the downstream RL tasks involving robotic object manipulations like hanging mugs on hooks, pushing objects, or opening doors. Video: https: //dannydriess. github. io/nerf-rl

IROS Conference 2022 Conference Paper

RHH-LGP: Receding Horizon And Heuristics-Based Logic-Geometric Programming For Task And Motion Planning

• Cornelius V. Braun
• Joaquim Ortiz de Haro
• Marc Toussaint
• Ozgur S. Oguz

Sequential decision-making and motion planning for robotic manipulation induce combinatorial complexity. For long-horizon tasks, especially when the environment comprises many objects that can be interacted with, planning efficiency becomes even more important. To plan such long-horizon tasks, we present the RHH-LGP algorithm for combined task and motion planning (TAMP). First, we propose a TAMP approach (based on Logic-Geometric Programming) that effectively uses geometry-based heuristics for solving long-horizon manipulation tasks. The efficiency of this planner is then further improved by a receding horizon formulation, resulting in RHH-LGP. We demonstrate the robustness and effectiveness of our approach on a diverse range of long-horizon tasks that require reasoning about interactions with a large number of objects. Using our framework, we can solve tasks that require multiple robots, including a mobile robot and snake-like walking robots, to form novel heterogeneous kinematic structures autonomously. By combining geometry-based heuristics with iterative planning, our approach brings an order-of-magnitude reduction of planning time in all investigated problems.

IROS Conference 2022 Conference Paper

Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation

• Marc Toussaint
• Jason Harris
• Jung-Su Ha
• Danny Driess
• Wolfgang Hönig

Task and Motion Planning has made great progress in solving hard sequential manipulation problems. However, a gap between such planning formulations and control methods for reactive execution remains. In this paper we pro-pose a model predictive control approach dedicated to robustly execute a single sequence of constraints, which corresponds to a discrete decision sequence of a TAMP plan. We decompose the overall control problem into three sub-problems (solving for sequential waypoints, their timing, and a short receding horizon path) that each is a non-linear program solved online in each MPC cycle. The resulting control strategy can account for long-term interdependencies of constraints and reactively plan for a timing-optimal transition through all constraints. We additionally propose phase backtracking when running constraints of the current phase cannot be fulfilled, leading to a fluent re-initiation behavior that is robust to perturbations and interferences by an experimenter.

ICRA Conference 2022 Conference Paper

ST-RRT*: Asymptotically-Optimal Bidirectional Motion Planning through Space-Time

• Francesco Grothe
• Valentin N. Hartmann
• Andreas Orthey
• Marc Toussaint

We present a motion planner for planning through space-time with dynamic obstacles, velocity constraints, and unknown arrival time. Our algorithm, Space-Time RRT*(ST-RRT*), is a probabilistically complete, bidirectional motion planning algorithm, which is asymptotically optimal with respect to the shortest arrival time. We experimentally evaluate ST-RRT* in both abstract (2D disk, 8D disk in cluttered spaces, and on a narrow passage problem), and simulated robotic path planning problems (sequential planning of 8DoF mobile robots, and 7DoF robotic arms). The proposed planner outperforms RRT-Connect and RRT* on both initial solution time, and attained final solution cost. The code for ST-RRT* is available in the Open Motion Planning Library (OMPL).

ICRA Conference 2021 Conference Paper

Co-Optimizing Robot, Environment, and Tool Design via Joint Manipulation Planning

• Marc Toussaint
• Jung-Su Ha
• Ozgur S. Oguz

Existing work on sequential manipulation planning and trajectory optimization typically assumes the robot, environment and tools to be given. However, in particular in industrial applications, it is highly interesting to ask, what would be an optimal robot design, tool shape, or robot station geometry for a particular ensemble of manipulation tasks. To tackle this problem we propose a formulation to jointly optimize over static design parameters and the sequential manipulation trajectory. We can include optimization objectives such as penalizing velocities (path length) and joint torques. Our evaluations show that design optimization can significantly improve on such metrics. For instance, in a wrench tool demonstration scenario we show that the shape of the wrench tool as well as design of the robot can be optimized to allow for exerting a necessary external torque with minimal effort.

ICRA Conference 2021 Conference Paper

Control-Tree Optimization: an approach to MPC under discrete Partial Observability

• Camille Phiquepal
• Marc Toussaint

This paper presents a new approach to Model Predictive Control for environments where essential, discrete variables are partially observed. Under this assumption, the belief state is a probability distribution over a finite number of states. We optimize a control-tree where each branch assumes a given state-hypothesis. The control-tree optimization uses the probabilistic belief state information. This leads to policies more optimized with respect to likely states than unlikely ones, while still guaranteeing robust constraint satisfaction at all times. We apply the method to both linear and non-linear MPC with constraints. The optimization of the control-tree is decomposed into optimization subproblems that are solved in parallel leading to good scalability for high number of state-hypotheses. We demonstrate the real-time feasibility of the algorithm on two examples and show the benefits compared to a classical MPC scheme optimizing w. r. t. one single hypothesis.

ICRA Conference 2021 Conference Paper

Deep 6-DoF Tracking of Unknown Objects for Reactive Grasping

• Marc Tuscher
• Julian Hörz
• Danny Driess
• Marc Toussaint

Robotic manipulation of unknown objects is an important field of research. Practical applications occur in many real-world settings where robots need to interact with an unknown environment. We tackle the problem of reactive grasping by proposing a method for unknown object tracking, grasp point sampling and dynamic trajectory planning. Our object tracking method combines Siamese Networks with an Iterative Closest Point approach for pointcloud registration into a method for 6-DoF unknown object tracking. The method does not require further training and is robust to noise and occlusion. We propose a robotic manipulation system, which is able to grasp a wide variety of formerly unseen objects and is robust against object perturbations and inferior grasping points.

ICRA Conference 2021 Conference Paper

Learning Efficient Constraint Graph Sampling for Robotic Sequential Manipulation

• Joaquim Ortiz de Haro
• Valentin N. Hartmann
• Ozgur S. Oguz
• Marc Toussaint

Efficient sampling from constraint manifolds, and thereby generating a diverse set of solutions for feasibility problems, is a fundamental challenge. We consider the case where a problem is factored, that is, the underlying nonlinear program is decomposed into differentiable equality and inequality constraints, each of which depends only on some variables. Such problems are at the core of efficient and robust sequential robot manipulation planning. Naive sequential conditional sampling of individual variables, as well as fully joint sampling of all variables at once (e. g. , leveraging optimization methods), can be highly inefficient and non-robust. We propose a novel framework to learn how to break the overall problem into smaller sequential sampling problems. Specifically, we leverage Monte-Carlo Tree Search to learn assignment orders for the variable-subsets, in order to minimize the computation time to generate feasible full samples. This strategy allows us to efficiently compute a set of diverse valid robot configurations for mode-switches within sequential manipulation tasks, which are waypoints for subsequent trajectory optimization or sampling-based motion planning algorithms. We show that the learning method quickly converges to the best sampling strategy for a given problem, and outperforms user-defined orderings or fully joint optimization, while providing a higher sample diversity. Video: https://youtu.be/mCNdvjTbHNI

ICRA Conference 2021 Conference Paper

Learning Geometric Reasoning and Control for Long-Horizon Tasks from Visual Input

• Danny Driess
• Jung-Su Ha
• Russ Tedrake
• Marc Toussaint

Long-horizon manipulation tasks require joint reasoning over a sequence of discrete actions and their associated continuous control parameters. While Task and Motion Planning (TAMP) approaches are capable of generating motion plans that account for this joint reasoning, they usually assume full knowledge about the environment (e. g. in terms of shapes, poses of objects) and often require computation times not suitable for real-time control. To overcome this, we propose a learning framework where a high-level reasoning network predicts, based on an image of the scene, a sequence of discrete actions and the parameter values of their associated low-level controllers. These controllers are parameterized in terms of a learned energy function, leading to time-invariant controllers for each phase. We train the whole framework end-to-end using a dataset of TAMP solutions computed using Logic Geometric Programming. A key feature is that the reasoning network determines the parameters of the controllers jointly, such that the overall task can be solved. Despite having no explicit representation of the geometry nor pose of the objects in the scene, our network is still able to accomplish geometrically precise manipulation tasks, including handovers and an accurate pointing task where the parameters of early actions are tightly coupled with those of later actions. Video: https://youtu.be/AcPWRTkr3_g

IROS Conference 2021 Conference Paper

Learning to Arbitrate Human and Robot Control using Disagreement between Sub-Policies

• Yoojin Oh
• Marc Toussaint
• Jim Mainprice

In the context of teleoperation, arbitration refers to deciding how to blend between human and autonomous robot commands. We present a reinforcement learning solution that learns an optimal arbitration strategy that allocates more control authority to the human when the robot comes across a decision point in the task. A decision point is where the robot encounters multiple options (sub-policies), such as having multiple paths to get around an obstacle or deciding between two candidate goals. By expressing each directional sub-policy as a von Mises distribution, we identify the decision points by observing the modality of the mixture distribution. Our reward function reasons on this modality and prioritizes to match its learned policy to either the user or the robot accordingly. We report teleoperation experiments on reach-and-grasping objects using a robot manipulator arm with different simulated human controllers. Results indicate that our shared control agent outperforms direct control and improves the teleoperation performance among different users. Using our reward term enables flexible blending between human and robot commands while maintaining safe and accurate teleoperation.

NeurIPS Conference 2021 Conference Paper

Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics

• Ingmar Schubert
• Danny Driess
• Ozgur S. Oguz
• Marc Toussaint

Applications of Reinforcement Learning (RL) in robotics are often limited by high data demand. On the other hand, approximate models are readily available in many robotics scenarios, making model-based approaches like planning a data-efficient alternative. Still, the performance of these methods suffers if the model is imprecise or wrong. In this sense, the respective strengths and weaknesses of RL and model-based planners are complementary. In the present work, we investigate how both approaches can be integrated into one framework that combines their strengths. We introduce Learning to Execute (L2E), which leverages information contained in approximate plans to learn universal policies that are conditioned on plans. In our robotic manipulation experiments, L2E exhibits increased performance when compared to pure RL, pure planning, or baseline methods combining learning and planning.

ICLR Conference 2021 Conference Paper

Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks

• Ingmar Schubert
• Ozgur S. Oguz
• Marc Toussaint

In high-dimensional state spaces, the usefulness of Reinforcement Learning (RL) is limited by the problem of exploration. This issue has been addressed using potential-based reward shaping (PB-RS) previously. In the present work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS). FV-RS relaxes the strict optimality guarantees of PB-RS to a guarantee of preserved long-term behavior. Being less restrictive, FV-RS allows for reward shaping functions that are even better suited for improving the sample efficiency of RL algorithms. In particular, we consider settings in which the agent has access to an approximate plan. Here, we use examples of simulated robotic manipulation tasks to demonstrate that plan-based FV-RS can indeed significantly improve the sample efficiency of RL over plan-based PB-RS.

ICRA Conference 2021 Conference Paper

Sparse Multilevel Roadmaps for High-Dimensional Robotic Motion Planning

• Andreas Orthey
• Marc Toussaint

Sparse roadmaps are important to compactly represent state spaces, to determine problems to be infeasible and to terminate in finite time. However, sparse roadmaps do not scale well to high-dimensional planning problems. In prior work, we showed improved planning performance on high-dimensional planning problems by using multilevel abstractions to simplify state spaces. In this work, we generalize sparse roadmaps to multilevel abstractions by developing a novel algorithm, the sparse multilevel roadmap planner (SMLR). To this end, we represent multilevel abstractions using the language of fiber bundles, and generalize sparse roadmap planners by using the concept of restriction sampling with visibility regions. We argue SMLR to be probabilistically complete and asymptotically near-optimal by inheritance from sparse roadmap planners. In evaluations, we outperform sparse roadmap planners on challenging planning problems, in particular problems which are high-dimensional, contain narrow passages or are infeasible. We thereby demonstrate sparse multilevel roadmaps as an efficient tool for feasible and infeasible high-dimensional planning problems.

ICRA Conference 2020 Conference Paper

A Probabilistic Framework for Constrained Manipulations and Task and Motion Planning under Uncertainty

• Jung-Su Ha
• Danny Driess
• Marc Toussaint

Logic-Geometric Programming (LGP) is a powerful motion and manipulation planning framework, which represents hierarchical structure using logic rules that describe discrete aspects of problems, e. g. , touch, grasp, hit, or push, and solves the resulting smooth trajectory optimization. The expressive power of logic allows LGP for handling complex, large-scale sequential manipulation and tool-use planning problems. In this paper, we extend the LGP formulation to stochastic domains. Based on the control-inference duality, we interpret LGP in a stochastic domain as fitting a mixture of Gaussians to the posterior path distribution, where each logic pro le defines a single Gaussian path distribution. The proposed framework enables a robot to prioritize various interaction modes and to acquire interesting behaviors such as contact exploitation for uncertainty reduction, eventually providing a composite control scheme that is reactive to disturbance.

ICRA Conference 2020 Conference Paper

Deep Visual Heuristics: Learning Feasibility of Mixed-Integer Programs for Manipulation Planning

• Danny Driess
• Ozgur S. Oguz
• Jung-Su Ha
• Marc Toussaint

In this paper, we propose a deep neural network that predicts the feasibility of a mixed-integer program from visual input for robot manipulation planning. Integrating learning into task and motion planning is challenging, since it is unclear how the scene and goals can be encoded as input to the learning algorithm in a way that enables to generalize over a variety of tasks in environments with changing numbers of objects and goals. To achieve this, we propose to encode the scene and the target object directly in the image space. Our experiments show that our proposed network generalizes to scenes with multiple objects, although during training only two objects are present at the same time. By using the learned network as a heuristic to guide the search over the discrete variables of the mixed-integer program, the number of optimization problems that have to be solved to find a feasible solution or to detect infeasibility can greatly be reduced.

ICRA Conference 2020 Conference Paper

Prediction of Human Full-Body Movements with Motion Optimization and Recurrent Neural Networks

• Philipp Kratzer
• Marc Toussaint
• Jim Mainprice

Human movement prediction is difficult as humans naturally exhibit complex behaviors that can change drastically from one environment to the next. In order to alleviate this issue, we propose a prediction framework that decouples short-term prediction, linked to internal body dynamics, and long-term prediction, linked to the environment and task constraints. In this work we investigate encoding short-term dynamics in a recurrent neural network, while we account for environmental constraints, such as obstacle avoidance, using gradient-based trajectory optimization. Experiments on real motion data demonstrate that our framework improves the prediction with respect to state-of-the-art motion prediction methods, as it accounts to beforehand unseen environmental structures. Moreover we demonstrate on an example, how this framework can be used to plan robot trajectories that are optimized to coordinate with a human partner.

IROS Conference 2020 Conference Paper

Robust Task and Motion Planning for Long-Horizon Architectural Construction Planning

• Valentin N. Hartmann
• Ozgur S. Oguz
• Danny Driess
• Marc Toussaint
• Achim Menges

Integrating robotic systems in architectural and construction processes is of core interest to increase the efficiency of the building industry. Automated planning for such systems enables design analysis tools and facilitates faster design iteration cycles for designers and engineers. However, generic task-and-motion planning (TAMP) for long-horizon construction processes is beyond the capabilities of current approaches. In this paper, we develop a multi-agent TAMP framework for long horizon problems such as constructing a full-scale building. To this end we extend the Logic-Geometric Programming framework by sampling-based motion planning, a limited horizon approach, and a task-specific structural stability optimization that allow an effective decomposition of the task. We show that our framework is capable of constructing a large pavilion built from several hundred geometrically unique building elements from start to end autonomously.

IROS Conference 2020 Conference Paper

Sample-Efficient Learning for Industrial Assembly using Qgraph-bounded DDPG

• Sabrina Hoppe
• Markus Giftthaler
• Robert Krug 0003
• Marc Toussaint

Recent progress in deep reinforcement learning has enabled agents to autonomously learn complex control strategies from scratch. Model-free approaches like Deep Deterministic Policy Gradients (DDPG) seem promising for applications with intricate dynamics, such as contact-rich manipulation tasks. However, these methods typically require large amounts of training data or meticulous hyperparameter tuning, limiting their usefulness for real-world robotics applications. In this paper, we evaluate and benchmark our recently proposed approach for improving model-free reinforcement learning with DDPG through Qgraph-based bounds in temporal difference learning. We directly apply the algorithm to a challenging real-world industrial insertion task and assess its performance (see https://youtu.be/Z_GcNbCWE-E).Empirical results show that the insertion task can be learned despite significant frictional forces and uncertainty, even in sparse-reward settings. We present an in-depth comparison based on a large number of experiments and demonstrate the advantages and performance of Qgraph-bounded DDPG: the learning process can be significantly sped up, robustified against bad choices of hyperparameters and runs with less memory requirements. Lastly, the presented results extend the current theoretical understanding of the link between data graph structure and soft divergence in DDPG.

IROS Conference 2019 Conference Paper

Active Inverse Model Learning with Error and Reachable Set Estimates

• Danny Driess
• Syn Schmitt
• Marc Toussaint

In this work, we propose a framework to learn an inverse model of redundant systems. We address three problems. By formalizing what it actually means to learn an inverse model, we derive a method where the inverse model, represented as a neural network, is learned by minimizing an upper bound on the real performance error, which is provided by a forward model (kernel regression or Gaussian process) learned on the currently available data. Most machine learning methods focus on learning the mapping of the function. For inverse models, it is, however, crucial to know the reachable set of the true forward model, since this becomes the domain of the inverse. Therefore, we secondly propose a method to estimate the reachable set of the system. Finally, we develop an active exploration strategy that is based on maximizing a lower bound on the true fill-distance to efficiently generate the data in the high dimensional input space. A key feature of our method is that the resulting learned inverse model provides error bounds on its performance. From an application point of view, this work is motivated by learning to control musculoskeletal systems. In the experiments, we show for both a simulated model of a human arm with six muscles and a real muscle-driven robot that the proposed method is able to learn the reachable set of these systems as well as a policy that enables to accurately control the position.

ICRA Conference 2019 Conference Paper

Active Multi-Contact Continuous Tactile Exploration with Gaussian Process Differential Entropy

• Danny Driess
• Daniel Hennes
• Marc Toussaint

In the present work, we propose an active tactile exploration framework to obtain a surface model of an unknown object utilizing multiple contacts simultaneously. To incorporate these multiple contacts, the exploration strategy is based on the differential entropy of the underlying Gaussian process implicit surface model, which formalizes the exploration with multiple contacts within an information theoretic context and additionally allows for nonmyopic multi-step planning. In contrast to many previous approaches, the robot continuously slides along the surface with its end-effectors to gather the tactile stimuli, instead of touching it at discrete locations. This is realized by closely integrating the surface model into the compliant controller framework. Furthermore, we extend our recently proposed sliding based tactile exploration approach to handle non-convex objects. In the experiments, it is shown that multiple contacts simultaneously leads to a more efficient exploration of complex, non-convex objects, not only in terms of time, but also with respect to the total moved distance of all end-effectors. Finally, we demonstrate our methodology with a real PR2 robot that explores an object with both of its arms.

IROS Conference 2019 Conference Paper

An Optimal Algorithm to Solve the Combined Task Allocation and Path Finding Problem

• Christian Henkel
• Jannik Abbenseth
• Marc Toussaint

We consider multi-agent transport task problems where, e. g. in a factory setting, items have to be delivered from a given start to a goal pose while the delivering robots need to avoid collisions with each other on the floor. We introduce a Task Conflict-Based Search (TCBS) Algorithm to solve the combined delivery task allocation and multiagent path planning problem optimally. The problem is known to be NP-hard and the optimal solver cannot scale. However, we introduce it as a baseline to evaluate the sub-optimality of other approaches. We show experimental results that compare our solver with different sub-optimal ones in terms of regret.

ICRA Conference 2019 Conference Paper

Combined Task and Motion Planning under Partial Observability: An Optimization-Based Approach

• Camille Phiquepal
• Marc Toussaint

We propose a novel approach to Combined Task and Motion Planning (TAMP) under partial observability. Previous optimization-based TAMP methods [1][2] compute optimal plans and paths assuming full observability. However, partial observability requires the solution to be a policy that reacts to the observations that the agent receives. We consider a formulation where observations introduce additional branching in the symbolic decision tree. The solution is now given by a reactive policy on the symbolic level together with a path tree that describes the branchings of optimal motion depending on the observations. Our method works in two stages: First, the symbolic policy is optimized using approximate path costs estimated from independent optimizations of trajectory pieces. Second, we fix the best symbolic policy and optimize a joint trajectory tree. We test our approach on object manipulation and autonomous driving examples. We also compare the algorithm's performance to a state-of-the-art TAMP planner in fully observable cases.

IJCAI Conference 2019 Conference Paper

Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning - Extended Abtract

• Marc Toussaint
• Kelsey R. Allen
• Kevin A. Smith
• Joshua B. Tenenbaum

We propose to formulate physical reasoning and manipulation planning as an optimization problem that integrates first order logic, which we call Logic-Geometric Programming.

ICML Conference 2019 Conference Paper

Trajectory-Based Off-Policy Deep Reinforcement Learning

• Andreas Doerr
• Michael Volpp
• Marc Toussaint
• Sebastian Trimpe
• Christian G. Daniel

Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining recent improvements in the reuse of off-policy data and exploration in parameter space with deterministic behavioral policies. The resulting objective is amenable to standard neural network optimization strategies like stochastic gradient descent or stochastic gradient Hamiltonian Monte Carlo. Incorporation of previous rollouts via importance sampling greatly improves data-efficiency, whilst stochastic optimization schemes facilitate the escape from local optima. We evaluate the proposed approach on a series of continuous control benchmark tasks. The results show that the proposed algorithm is able to successfully and reliably learn solutions using fewer system interactions than standard policy gradient methods.

AAAI Conference 2018 Conference Paper

Bayesian Functional Optimization

• Ngo Anh Vien
• Heiko Zimmermann
• Marc Toussaint

Bayesian optimization (BayesOpt) is a derivative-free approach for sequentially optimizing stochastic black-box functions. Standard BayesOpt, which has shown many successes in machine learning applications, assumes a finite dimensional domain which often is a parametric space. The parameter space is defined by the features used in the function approximations which are often selected manually. Therefore, the performance of BayesOpt inevitably depends on the quality of chosen features. This paper proposes a new Bayesian optimization framework that is able to optimize directly on the domain of function spaces. The resulting framework, Bayesian Functional Optimization (BFO), not only extends the application domains of BayesOpt to functional optimization problems but also relaxes the performance dependency on the chosen parameter space. We model the domain of functions as a reproducing kernel Hilbert space (RKHS), and use the notion of Gaussian processes on a real separable Hilbert space. As a result, we are able to define traditional improvement-based (PI and EI) and optimistic acquisition functions (UCB) as functionals. We propose to optimize the acquisition functionals using analytic functional gradients that are also proved to be functions in a RKHS. We evaluate BFO in three typical functional optimization tasks: i) a synthetic functional optimization problem, ii) optimizing activation functions for a multi-layer perceptron neural network, and iii) a reinforcement learning task whose policies are modeled in RKHS.

IROS Conference 2018 Conference Paper

Kinematic Morphing Networks for Manipulation Skill Transfer

• Peter Englert
• Marc Toussaint

The transfer of a robot skill between different geometric environments is non-trivial since a wide variety of environments exists, sensor observations as well as robot motions are high-dimensional, and the environment might only be partially observed. We consider the problem of extracting a low-dimensional description of the manipulated environment in form of a kinematic model. This allows us to transfer a skill by defining a policy on a prototype model and morphing the observed environment to this prototype. A deep neural network is used to map depth image observations of the environment to morphing parameter, which include transformations and configurations of the prototype model. Using the concatenation property of affine transformations and the ability to convert point clouds to depth images allows to apply the network in an iterative manner. The network is trained on data generated in a simulator and on augmented data that is created with its own predictions. The algorithm is evaluated on different tasks, where it is shown that iterative predictions lead to a higher accuracy than one-step predictions.

ICRA Conference 2018 Conference Paper

Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties

• Danny Driess
• Heiko Zimmermann
• Simon Wolfen
• Dan Suissa
• Daniel F. B. Häufle
• Daniel Hennes
• Marc Toussaint
• Syn Schmitt

Modeling biomechanical musculoskeletal systems reveals that the mapping from muscle stimulations to movement dynamics is highly nonlinear and complex, which makes it difficult to control those systems with classical techniques. In this work, we not only investigate whether machine learning approaches are capable of learning a controller for such systems. We are especially interested in the question if the structure of the musculoskeletal apparatus exhibits properties that are favorable for the learning task. In particular, we consider learning a control policy from target positions to muscle stimulations. To account for the high actuator redundancy of biomechanical systems, our approach uses a learned forward model represented by a neural network and sequential quadratic programming to obtain the control policy, which also enables us to alternate the co-contraction level and hence allows to change the stiffness of the system and to include optimality criteria like small muscle stimulations. Experiments on both a simulated musculoskeletal model of a human arm and a real biomimetic muscle-driven robot show that our approach is able to learn an accurate controller despite high redundancy and nonlinearity, while retaining sample efficiency.

ICRA Conference 2018 Conference Paper

Planning Ergonomic Sequences of Actions in Human-Robot Interaction

• Baptiste Busch
• Marc Toussaint
• Manuel Lopes 0001

In this paper, we define the problem of human-robot collaboration as a combined task and motion planning problem which is extended to the multi-agent case (human and robot). Our proposed approach allows us to explicitly take into account ergonomic cost, synchrony and concurrency of behavior in an optimization formulation. We show simulated results as well as an experiment with a real robot combined with a user study. Results show that optimizing over a sequence of actions leads to more ergonomic situations.

ICML Conference 2018 Conference Paper

Probabilistic Recurrent State-Space Models

• Andreas Doerr
• Christian G. Daniel
• Martin Schiegg
• Duy Nguyen-Tuong
• Stefan Schaal
• Marc Toussaint
• Sebastian Trimpe

State-space models (SSMs) are a highly expressive model class for learning patterns in time series data and for system identification. Deterministic versions of SSMs (e. g. , LSTMs) proved extremely successful in modeling complex time series data. Fully probabilistic SSMs, however, are often found hard to train, even for smaller problems. We propose a novel model formulation and a scalable training algorithm based on doubly stochastic variational inference and Gaussian processes. This combination allows efficient incorporation of latent state temporal correlations, which we found to be key to robust training. The effectiveness of the proposed PR-SSM is evaluated on a set of real-world benchmark datasets in comparison to state-of-the-art probabilistic model learning methods. Scalability and robustness are demonstrated on a high dimensional problem.

IROS Conference 2017 Conference Paper

Active learning with query paths for tactile object shape exploration

• Danny Driess
• Peter Englert
• Marc Toussaint

In the present work, we propose an active learning framework based on optimal query paths to efficiently address the problem of tactile object shape exploration. Most previous approaches perform active touch probing at discrete query points, which leads to inefficient touch-and-retract motions. In contrast, in this paper we propose to query information efficient sliding paths instead of only touch locations. This is realized by three components: A Gaussian process implicit surface model represents the shape and uncertainty of the object. A compliant task/force controller framework fuses the information of this GP model into the parameterization of its tasks, which enables the robot to slide over the unknown object safely and robustly. Thirdly, we develop two strategies to solve the proposed active path querying learning problem. Sliding along those query paths not only creates more dense data than touch probing, but additionally greatly reduces the uncertainty of the object. We demonstrate the effectiveness of our proposed framework both in simulation and on the PR2 robot platform. Furthermore, it is shown that our methodology can be extended to other learning tasks, such as finding a desired surface normal on an unknown object, e. g. for pushing.

ICRA Conference 2017 Conference Paper

Constrained Bayesian optimization of combined interaction force/task space controllers for manipulations

• Danny Driess
• Peter Englert
• Marc Toussaint

In this paper, we address the problem of how a robot can optimize parameters of combined interaction force/task space controllers under a success constraint in an active way. To enable the robot to explore its environment robustly, safely and without the risk of damaging anything, suitable control concepts have to be developed that enable compliant and force control in situations that are afflicted with high uncertainties. Instances of such concepts are impedance, operational space or hybrid control. However, the parameters of these controllers have to be tuned precisely in order to achieve reasonable performance, which is inherently challenging, as often no sufficient model of the environment is available. To overcome this, we propose to use constrained Bayesian optimization to enable the robot to tune its controller parameters autonomously. Unlike other controller tuning methods, this method allows us to include a success constraint into the optimization. Further, we introduce novel performance measures for compliant, force controlled robots. In real world experiments we show that our approach is able to optimize the parameters for a task that consists of establishing and maintaining contact between the robot and the environment efficiently and successfully.

ICRA Conference 2017 Conference Paper

Multi-bound tree search for logic-geometric programming in cooperative manipulation domains

• Marc Toussaint
• Manuel Lopes 0001

Joint symbolic and geometric planning is one of the core challenges in robotics. We address the problem of multi-agent cooperative manipulation, where we aim for jointly optimal paths for all agents and over the full manipulation sequence. This joint optimization problem can be framed as a logic-geometric program. Existing solvers lack several features (such as consistently handling kinematic switches) and efficiency to handle the cooperative manipulation domain. We propose a new approximate solver scheme, combining ideas from branch-and-bound and MCTS and exploiting multiple levels of bounds to better direct the search. We demonstrate the method in a scenario where a Baxter robot needs to help a human to reach for objects.

ICRA Conference 2017 Conference Paper

Preference learning on the execution of collaborative human-robot tasks

• Thibaut Munzer
• Marc Toussaint
• Manuel Lopes 0001

We present a novel method to learn human preferences during, and for, the execution of concurrent joint humanrobot tasks. We consider tasks realized by a team of a human operator and a robot helper that should adapt to the human's task execution preferences. Different human operators can have different abilities, experiences, and personal preferences, so that a particular allocation of activities in the team is preferred over another. We cast the behavior of concurrent multi-agent cooperation as a semi Markov Decision Process and show how to model and learn human preferences over the team behavior. After proposing two different interactive learning algorithms, we evaluate them and show that the system can effectively learn and adapt to human preferences.

IJCAI Conference 2016 Conference Paper

Policy Search in Reproducing Kernel Hilbert Space

• Ngo Anh Vien
• Peter Englert
• Marc Toussaint

Modeling policies in reproducing kernel Hilbert space (RKHS) renders policy gradient reinforcement learning algorithms non-parametric. As a result, the policies become very flexible and have a rich representational potential without a pre-defined set of features. However, their performances might be either non-covariant under re-parameterization of the chosen kernel, or very sensitive to step-size selection. In this paper, we propose to use a general framework to derive a new RKHS policy search technique. The new derivation leads to both a natural RKHS actor-critic algorithm and a RKHS expectation maximization (EM) policy search algorithm. Further, we show that kernelization enables us to learn in partially observable (POMDP) tasks which is considered daunting for parametric approaches. Via sparsification, a small set of "support vectors" representing the history is shown to be effectively discovered. For evaluations, we use three simulated (PO)MDP reinforcement learning tasks, and a simulated PR2's robotic manipulation task. The results demonstrate the effectiveness of the new RKHS policy search framework in comparison to plain RKHS actor-critic, episodic natural actor-critic, plain actor-critic, and PoWER approaches.

ICRA Conference 2016 Conference Paper

Relational activity processes for modeling concurrent cooperation

• Marc Toussaint
• Thibaut Munzer
• Yoan Mollard
• Li Yang Wu
• Ngo Anh Vien
• Manuel Lopes 0001

In human-robot collaboration, multi-agent domains, or single-robot manipulation with multiple end-effectors, the activities of the involved parties are naturally concurrent. Such domains are also naturally relational as they involve objects, multiple agents, and models should generalize over objects and agents. We propose a novel formalization of relational concurrent activity processes that allows us to transfer methods from standard relational MDPs, such as Monte-Carlo planning and learning from demonstration, to concurrent cooperation domains. We formally compare the formulation to previous propositional models of concurrent decision making and demonstrate planning and learning from demonstration methods on a real-world human-robot assembly task.

ICRA Conference 2015 Conference Paper

Active exploration of joint dependency structures

• Johannes Kulick
• Stefan Otte
• Marc Toussaint

Being able to manipulate degrees of freedom of the environment, such as doors or drawers, is a requirement for most tasks a robot is supposed to perform. Often these external degrees of freedom depend on other ones, e. g. , a drawer can only be opened if the lock is not locking the joint. We propose an approach to autonomously and efficiently explore and uncover joint dependency structures. We develop a probabilistic model for joint dependency structures which is the basis for active learning. Discontinuities in the dynamics of the joint, which often indicate key points of the joint, are used to segment the joint space into meaningful segments which then allows efficient exploration with the developed maximum cross-entropy (MaxCE) exploration strategy. Experiments in a simulated environment and on a real PR2 suggest that the proposed approach yields efficient exploration of joint dependency structures.

IROS Conference 2015 Conference Paper

Automatic testing and minimax optimization of system parameters for best worst-case performance

• Kim Peter Wabersich
• Marc Toussaint

Robotic systems typically have numerous parameters, e. g. the choice of planning algorithm, real-valued parameters of motion and vision modules, and control parameters. We consider the problem of optimizing these parameters for best worst-case performance over a range of environments. To this end we first propose to evaluate system parameters by adversarially optimizing over environment parameters to find particularly hard environments. This is then nested in a game-theoretic minimax optimization setting, where an outerloop aims to find best worst-case system parameters. For both optimization levels we use Bayesian global optimization (GP-UCB) which provides the necessary confidence bounds to handle the stochasticity of the performance. We compare our method (Nested Minimax) with an existing relaxation method we adapted to become applicable in our setting. By construction our approach provides more robustness to performance stochasticity. We demonstrate the method for planning algorithm selection on a pick'n'place application and for control parameter optimization on a triple inverted pendulum for robustness to adversarial perturbations.

AAAI Conference 2015 Conference Paper

Hierarchical Monte-Carlo Planning

• Ngo Anh Vien
• Marc Toussaint

Monte-Carlo Tree Search, especially UCT and its POMDP version POMCP, have demonstrated excellent performance on many problems. However, to efficiently scale to large domains one should also exploit hierarchical structure if present. In such hierarchical domains, finding rewarded states typically requires to search deeply; covering enough such informative states very far from the root becomes computationally expensive in flat non-hierarchical search approaches. We propose novel, scalable MCTS methods which integrate a task hierarchy into the MCTS framework, specifically leading to hierarchical versions of both, UCT and POMCP. The new method does not need to estimate probabilistic models of each subtask, it instead computes subtask policies purely sample-based. We evaluate the hierarchical MCTS methods on various settings such as a hierarchical MDP, a Bayesian model-based hierarchical RL problem, and a large hierarchical POMDP.

IJCAI Conference 2015 Conference Paper

Logic-Geometric Programming: An Optimization-Based Approach to Combined Task and Motion Planning

• Marc Toussaint

We consider problems of sequential robot manipulation (aka. combined task and motion planning) where the objective is primarily given in terms of a cost function over the final geometric state, rather than a symbolic goal description. In this case we should leverage optimization methods to inform search over potential action sequences. We propose to formulate the problem holistically as a 1storder logic extension of a mathematical program: a non-linear constrained program over the full world trajectory where the symbolic state-action sequence defines the (in-)equality constraints. We tackle the challenge of solving such programs by proposing three levels of approximation: The coarsest level introduces the concept of the effective end state kinematics, parametrically describing all possible end state configurations conditional to a given symbolic action sequence. Optimization on this level is fast and can inform symbolic search. The other two levels optimize over interaction keyframes and eventually over the full world trajectory across interactions. We demonstrate the approach on a problem of maximizing the height of a physically stable construction from an assortment of boards, cylinders and blocks.

IROS Conference 2015 Conference Paper

POMDP manipulation via trajectory optimization

• Ngo Anh Vien
• Marc Toussaint

Efficient object manipulation based only on force feedback typically requires a plan of actively contact-seeking actions to reduce uncertainty over the true environmental model. In principle, that problem could be formulated as a full partially observable Markov decision process (POMDP) whose observations are sensed forces indicating the presence/absence of contacts with objects. Such a naive application leads to a very large POMDP with high-dimensional continuous state, action and observation spaces. Solving such large POMDPs is practically prohibitive. In other words, we are facing three challenging problems: 1) uncertainty over discontinuous contacts with objects; 2) high-dimensional continuous spaces; 3) optimization for not only trajectory cost but also execution time. As trajectory optimization is a powerful model-based method for motion generation, it can handle the last two issues effectively by computing locally optimal trajectories. This paper aims to integrate advantages of trajectory optimization into existing POMDP solvers. The full POMDP formulation is solved using sample-based approaches, where each sampled model is quickly evaluated via trajectory optimization instead of simulating a large number of rollouts. To further accelerate the solver, we propose to integrate temporal abstraction, i. e. macro actions or temporal actions, into the POMDP model. We demonstrate the proposed method on a simulated 7 DoF KUKA arm and a physical Willow Garage PR2 platform. The results show that our proposed method could effectively seek contacts in complex scenarios, and achieve near-optimal performance of path planing.

JAIR Journal 2015 Journal Article

Probabilistic Inference Techniques for Scalable Multiagent Decision Making

• Akshat Kumar
• Shlomo Zilberstein
• Marc Toussaint

Decentralized POMDPs provide an expressive framework for multiagent sequential decision making. However, the complexity of these models---NEXP-Complete even for two agents---has limited their scalability. We present a promising new class of approximation algorithms by developing novel connections between multiagent planning and machine learning. We show how the multiagent planning problem can be reformulated as inference in a mixture of dynamic Bayesian networks (DBNs). This planning-as-inference approach paves the way for the application of efficient inference techniques in DBNs to multiagent decision making. To further improve scalability, we identify certain conditions that are sufficient to extend the approach to multiagent systems with dozens of agents. Specifically, we show that the necessary inference within the expectation-maximization framework can be decomposed into processes that often involve a small subset of agents, thereby facilitating scalability. We further show that a number of existing multiagent planning models satisfy these conditions. Experiments on large planning benchmarks confirm the benefits of our approach in terms of runtime and scalability with respect to existing techniques.

IROS Conference 2015 Conference Paper

Robot programming from demonstration, feedback and transfer

• Yoan Mollard
• Thibaut Munzer
• Andrea Baisero
• Marc Toussaint
• Manuel Lopes 0001

This paper presents a novel approach for robot instruction for assembly tasks. We consider that robot programming can be made more efficient, precise and intuitive if we leverage the advantages of complementary approaches such as learning from demonstration, learning from feedback and knowledge transfer. Starting from low-level demonstrations of assembly tasks, the system is able to extract a high-level relational plan of the task. A graphical user interface (GUI) allows then the user to iteratively correct the acquired knowledge by refining high-level plans, and low-level geometrical knowledge of the task. This combination leads to a faster programming phase, more precise than just demonstrations, and more intuitive than just through a GUI. A final process allows to reuse high-level task knowledge for similar tasks in a transfer learning fashion. Finally we present a user study illustrating the advantages of this approach.

ICRA Conference 2015 Conference Paper

Sparse Gaussian process regression for compliant, real-time robot control

• Jens Schreiter
• Peter Englert
• Duy Nguyen-Tuong
• Marc Toussaint

Sparse Gaussian process (GP) models provide an efficient way to perform regression on large data sets. The key idea is to select a representative subset of the available training data, which induces the sparse GP model approximation. In the past, a variety of selection criteria for GP approximation have been proposed, but they either lack accuracy or suffer from high computational costs. In this paper, we introduce a novel and straightforward criterion for successive selection of training points used for GP model approximation. The proposed algorithm allows a fast and efficient selection of training points, while being competitive in learning performance. As evaluation, we employ our approach in learning inverse dynamics models for robot control using very large data sets (e. g. 500. 000 samples). It is demonstrated in experiments that our approximated GP model is sufficiently fast for real-time prediction in robot control. Comparisons with other state-of-the-art approximation techniques show that our proposed approach is significantly faster, while being competitive to generalization accuracy.

IROS Conference 2015 Conference Paper

Temporal segmentation of pair-wise interaction phases in sequential manipulation demonstrations

• Andrea Baisero
• Yoan Mollard
• Manuel Lopes 0001
• Marc Toussaint
• Ingo Lütkebohle

We consider the problem of learning from complex sequential demonstrations. We propose to analyze demonstrations in terms of the concurrent interaction phases which arise between pairs of involved bodies (hand-object and object-object). These interaction phases are the key to decompose a full demonstration into its atomic manipulation actions and to extract their respective consequences. In particular, one may assume that the goal of each interaction phase is to achieve specific geometric constraints between objects. This generalizes previous Learning from Demonstration approaches by considering not just the motion of the end-effector but also the relational properties of the objects' motion. We present a linear-chain Conditional Random Field model to detect the pair-wise interaction phases and extract the geometric constraints that are established in the environment, which represent a high-level task oriented description of the demonstrated manipulation. We test our system on single- and multi-agent demonstrations of assembly tasks, respectively of a wooden toolbox and a plastic chair.

ICRA Conference 2015 Conference Paper

Understanding the geometry of workspace obstacles in Motion Optimization

• Nathan D. Ratliff
• Marc Toussaint
• Stefan Schaal

What is it that makes movement around obstacles hard? The answer seems clear: obstacles contort the geometry of the workspace and make it difficult to leverage what we consider easy and intuitive straight-line Cartesian geometry. But is Cartesian motion actually easy? It's certainly well-understood and has numerous applications. But beneath the details of linear algebra and pseudoinverses, lies a non-trivial Riemannian metric driving the solution. Cartesian motion is easy only because the pseudoinverse, our powerhouse tool, correctly represents how Euclidean workspace geometry pulls back into the configuration space. In light of that observation, it reasons that motion through a field of obstacles could be just as easy as long as we correctly account for how those obstacles warp the geometry of the space. This paper explores extending our geometric model of the robot beyond the notion of a Cartesian workspace space to fully model and leverage how geometry changes in the presence of obstacles. Intuitively, impenetrable obstacles form topological holes and geodesics curve around them accordingly. We formalize this intuition and develop a general motion optimization framework called Riemannian Motion Optimization (RieMO) to efficiently find motions using our geometric models. Our experiments demonstrate that, for many problems, obstacle avoidance can be much more natural when placed within the right geometric context.

IROS Conference 2014 Conference Paper

Dual execution of optimized contact interaction trajectories

• Marc Toussaint
• Nathan D. Ratliff
• Jeannette Bohg
• Ludovic Righetti
• Peter Englert
• Stefan Schaal

Efficient manipulation requires contact to reduce uncertainty. The manipulation literature refers to this as funneling: a methodology for increasing reliability and robustness by leveraging haptic feedback and control of environmental interaction. However, there is a fundamental gap between traditional approaches to trajectory optimization and this concept of robustness by funneling: traditional trajectory optimizers do not discover force feedback strategies. From a POMDP perspective, these behaviors could be regarded as explicit observation actions planned to sufficiently reduce uncertainty thereby enabling a task. While we are sympathetic to the full POMDP view, solving full continuous-space POMDPs in high-dimensions is hard. In this paper, we propose an alternative approach in which trajectory optimization objectives are augmented with new terms that reward uncertainty reduction through contacts, explicitly promoting funneling. This augmentation shifts the responsibility of robustness toward the actual execution of the optimized trajectories. Directly tracing trajectories through configuration space would lose all robustness—dual execution achieves robustness by devising force controllers to reproduce the temporal interaction profile encoded in the dual solution of the optimization problem. This work introduces dual execution in depth and analyze its performance through robustness experiments in both simulation and on a real-world robotic platform.

IROS Conference 2014 Conference Paper

Entropy-based strategies for physical exploration of the environment's degrees of freedom

• Stefan Otte
• Johannes Kulick
• Marc Toussaint
• Oliver Brock

Physical exploration refers to the challenge of autonomously discovering and learning how to manipulate the environment's degrees of freedom (DOF)-by identifying promising points of interaction and pushing or pulling object parts to reveal DOF and their properties. Recent existing work focused on sub-problems like estimating DOF parameters from given data. Here, we address the integrated problem, focusing on the higher-level strategy to iteratively decide on the next exploration point before applying motion generation methods to execute the explorative action and data analysis methods to interpret the feedback. We propose to decide on exploration points based on the expected information gain, or change in entropy in the robot's current belief (uncertain knowledge) about the DOF. To this end, we first define how we represent such a belief. This requires dealing with the fact that the robot initially does not know which random variables (which DOF, and depending on their type, which DOF properties) actually exist. We then propose methods to estimate the expected information gain for an exploratory action. We analyze these strategies in simple environments and evaluate them in combination with full motion planning and data analysis in a physical simulation environment.

ICML Conference 2014 Conference Paper

Model-Based Relational RL When Object Existence is Partially Observable

• Ngo Anh Vien
• Marc Toussaint

We consider learning and planning in relational MDPs when object existence is uncertain and new objects may appear or disappear depending on previous actions or properties of other objects. Optimal policies actively need to discover objects to achieve a goal; planning in such domains in general amounts to a POMDP problem, where the belief is about the existence and properties of potential not-yet-discovered objects. We propose a computationally efficient extension of model-based relational RL methods that approximates these beliefs using discrete uncertainty predicates. In this formulation the belief update is learned using probabilistic rules and planning in the approximated belief space can be achieved using an extension of existing planners. We prove that the learned belief update rules encode an approximation of the exact belief updates of a POMDP formulation and demonstrate experimentally that the proposed approach successfully learns a set of relational rules appropriate to solve such problems.

IROS Conference 2014 Conference Paper

Reactive phase and task space adaptation for robust motion execution

• Peter Englert
• Marc Toussaint

An essential aspect for making robots succeed in real-world environments is to give them the ability to robustly perform motions in continuously changing situations. Classical motion planning methods usually create plans for static environments. The direct execution of such plans in dynamic environments often becomes problematic. We present an approach that adapts motion plans by feeding changes of the environment into a transformation of the plan in task space. Furthermore, the progress in the plan is defined with a phase variable that is updated adaptively according to the actual task progress. This phase variable releases the strict time compliance that many motion planning methods bring along. The main benefit of our approach is the ability to do this adaptation in a computational efficient manner during the execution of the motion. Thus, the gap between the motion planning and motion execution stage is bridged by continuously transforming geometric and dynamic features of a reference plan to the current situation. We evaluate the performance of our approach by comparing it to alternative methods such as dynamic motion primitives and continuous replanning on several simulated benchmark tasks. Moreover, we demonstrate the real robot applicability on a PR2 robot platform.

IJCAI Conference 2013 Conference Paper

Active Learning for Teaching a Robot Grounded Relational Symbols

• Johannes Kulick
• Marc Toussaint
• Tobias Lang
• Manuel Lopes

We investigate an interactive teaching scenario, where a human teaches a robot symbols which abstract the geometric properties of objects. There are multiple motivations for this scenario: First, state-of-the-art methods for relational reinforcement learning demonstrate that we can learn and employ strongly generalizing abstract models with great success for goal-directed object manipulation. However, these methods rely on given grounded action and state symbols and raise the classical question: Where do the symbols come from? Second, existing research on learning from humanrobot interaction has focused mostly on the motion level (e. g. , imitation learning). However, if the goal of teaching is to enable the robot to autonomously solve sequential manipulation tasks in a goal-directed manner, the human should have the possibility to teach the relevant abstractions to describe the task and let the robot eventually leverage powerful relational RL methods. In this paper we formalize human-robot teaching of grounded symbols as an active learning problem, where the robot actively generates pick-and-place geometric situations that maximize its information gain about the symbol to be learned. We demonstrate that the learned symbols can be used by a robot in a relational RL framework to learn probabilistic relational rules and use them to solve object manipulation tasks in a goal-directed manner.

IROS Conference 2013 Conference Paper

Caging complex objects with geodesic balls

• Dmitry Zarubin
• Florian T. Pokorny
• Marc Toussaint
• Danica Kragic

This paper proposes a novel approach for the synthesis of grasps of objects whose geometry can be observed only in the presence of noise. We focus in particular on the problem of generating caging grasps with a realistic robot hand simulation and show that our method can generate such grasps even on complex objects. We introduce the idea of using geodesic balls on the object's surface in order to approximate the maximal contact surface between a robotic hand and an object. We define two types of heuristics which extract information from approximate geodesic balls in order to identify areas on an object that can likely be used to generate a caging grasp. Our heuristics are based on two scoring functions. The first uses winding angles measuring how much a geodesic ball on the surface winds around a dominant axis, while the second explores using the total discrete Gaussian curvature of a geodesic ball to rank potential caging postures. We evaluate our approach with respect to variations in hand kinematics, for a selection of complex real-world objects and with respect to its robustness to noise.

IJCAI Conference 2013 Conference Paper

On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference (Extended Abstract)

• Konrad Rawlik
• Marc Toussaint
• Sethu Vijayakumar

We present a reformulation of the stochastic optimal control problem in terms of KL divergence minimisation, not only providing a unifying perspective of previous approaches in this area, but also demonstrating that the formalism leads to novel practical approaches to the control problem. Specifically, a natural relaxation of the dual formulation gives rise to exact iterative solutions to the finite and infinite horizon stochastic optimal control problem, while direct application of Bayesian inference methods yields instances of risk sensitive control.

ICRA Conference 2013 Conference Paper

Optimizing motion primitives to make symbolic models more predictive

• Andreas Orthey
• Marc Toussaint
• Nikolay Jetchev

Solving complex robot manipulation tasks requires to combine motion generation on the geometric level with planning on a symbolic level. On both levels robotics research has developed a variety of mature methodologies, including geometric motion planning and motion primitive learning on the motor level as well as logic reasoning and relational Reinforcement Learning methods on the symbolic level. However, their robust integration remains a great challenge. In this paper we approach one aspect of this integration by optimizing the motion primitives on the geometric level to be as consistent as possible with their symbolic predictions. The so optimized motion primitives increase the probability of a “successful” motion-meaning that the symbolic prediction was indeed achieved. Conversely, using these optimized motion primitives to collect new data about the effects of actions the learnt symbolic rules becomes more predictive and deterministic.

IJCAI Conference 2013 Conference Paper

Path Integral Control by Reproducing Kernel Hilbert Space Embedding

• Konrad Rawlik
• Marc Toussaint
• Sethu Vijayakumar

We present an embedding of stochastic optimal control problems, of the so called path integral form, into reproducing kernel Hilbert spaces. Using consistent, sample based estimates of the embedding leads to a model-free, non-parametric approach for calculation of an approximate solution to the control problem. This formulation admits a decomposition of the problem into an invariant and task dependent component. Consequently, we make much more efficient use of the sample data compared to previous sample based approaches in this domain, e. g. , by allowing sample re-use across tasks. Numerical examples on test problems, which illustrate the sample efficiency, are provided.

EWRL Workshop 2013 Workshop Paper

Temporal Abstraction in Reinforcement Learning with Proximity Statistics

• Rico Jonschkowski
• Marc Toussaint

Automatic discovery of temporal abstractions is a key problem in hierarchical reinforcement learning. We propose a new approach to this problem by learning a time marginalized transition probability which we call proximity statistics from successful trajectories for related tasks. Viewing the proximity statistics as state values allows the agent to generate greedy policies from them. Making the statistics sparse and combining proximity estimates by proximity propagation can substantially accelerate planning compared to value iteration while keeping the size of the statistics manageable.

ICRA Conference 2013 Conference Paper

Uncertainty aware grasping and tactile exploration

• Stanimir Dragiev
• Marc Toussaint
• Michael Gienger

The perception of the surrounding world depends on noisy sensors which introduce uncertainty. When we develop algorithms for grasping with robotic hands it is not enough to assume the best estimate of the environment - if there is a measure of uncertainty we need to account for it. This paper presents a control law which augments a grasp controller with the ability to prefer known or unseen regions of an object; this leads to the introduction of two motion primitives: an explorative and exploitative grasp. We integrate this control law in a framework for iterative grasping and implement a tactile exploration scenario. The experimental results confirm that using the notion of uncertainty in the control loop yields better models and does it faster than an uninformed controller.

NeurIPS Conference 2012 Conference Paper

Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress

• Manuel Lopes
• Tobias Lang
• Marc Toussaint
• Pierre-Yves Oudeyer

Formal exploration approaches in model-based reinforcement learning estimate the accuracy of the currently learned model without consideration of the empirical prediction error. For example, PAC-MDP approaches such as Rmax base their model certainty on the amount of collected data, while Bayesian approaches assume a prior over the transition dynamics. We propose extensions to such approaches which drive exploration solely based on empirical estimates of the learner's accuracy and learning progress. We provide a ``sanity check'' theoretical analysis, discussing the behavior of our extensions in the standard stationary finite state-action case. We then provide experimental studies demonstrating the robustness of these exploration measures in cases of non-stationary environments or where original approaches are misled by wrong domain assumptions.

JMLR Journal 2012 Journal Article

Exploration in Relational Domains for Model-based Reinforcement Learning

• Tobias Lang
• Marc Toussaint
• Kristian Kersting

A fundamental problem in reinforcement learning is balancing exploration and exploitation. We address this problem in the context of model-based reinforcement learning in large stochastic relational domains by developing relational extensions of the concepts of the E 3 and R-MAX algorithms. Efficient exploration in exponentially large state spaces needs to exploit the generalization of the learned model: what in a propositional setting would be considered a novel situation and worth exploration may in the relational setting be a well-known context in which exploitation is promising. To address this we introduce relational count functions which generalize the classical notion of state and action visitation counts. We provide guarantees on the exploration efficiency of our framework using count functions under the assumption that we had a relational KWIK learner and a near-optimal planner. We propose a concrete exploration algorithm which integrates a practically efficient probabilistic rule learner and a relational planner (for which there are no guarantees, however) and employs the contexts of learned relational rules as features to model the novelty of states and actions. Our results in noisy 3D simulated robot manipulation problems and in domains of the international planning competition demonstrate that our approach is more effective than existing propositional and factored exploration techniques. [abs] [ pdf ][ bib ] &copy JMLR 2012. ( edit, beta )

AAMAS Conference 2011 Conference Paper

Escaping Local Optima in POMDP Planning as Inference

• Pascal Poupart
• Tobias Lang
• Marc Toussaint

Planning as inference recently emerged as a versatile approach to decision-theoretic planning and reinforcement learning for single and multi-agent systems in fully and partially observable domains with discrete and continuous variables. Since planning as inference essentially tackles a non-convex optimization problem when the states are partially observable, there is a need to develop techniques that can robustly escape local optima. We propose two algorithms: the first one adds nodes to the controller according to an increasingly deep forward search, while the second one splits nodes in a greedy fashion to improve reward likelihood.

ICRA Conference 2011 Conference Paper

Gaussian process implicit surfaces for shape estimation and grasping

• Stanimir Dragiev
• Marc Toussaint
• Michael Gienger

The choice of an adequate object shape representation is critical for efficient grasping and robot manipulation. A good representation has to account for two requirements: it should allow uncertain sensory fusion in a probabilistic way and it should serve as a basis for efficient grasp and motion generation. We consider Gaussian process implicit surface potentials as object shape representations. Sensory observations condition the Gaussian process such that its posterior mean defines an implicit surface which becomes an estimate of the object shape. Uncertain visual, haptic and laser data can equally be fused in the same Gaussian process shape estimate. The resulting implicit surface potential can then be used directly as a basis for a reach and grasp controller, serving as an attractor for the grasp end-effectors and steering the orientation of contact points. Our proposed controller results in a smooth reach and grasp trajectory without strict separation of phases. We validate the shape estimation using Gaussian processes in a simulation on randomly sampled shapes and the grasp controller on a real robot with 7DoF arm and 7DoF hand.

IJCAI Conference 2011 Conference Paper

Scalable Multiagent Planning Using Probabilistic Inference

• Akshat Kumar
• Shlomo Zilberstein
• Marc Toussaint

Multiagent planning has seen much progress with the development of formal models such as Dec-POMDPs. However, the complexity of these models-NEXP-Complete even for two agents-has limited scalability. We identify certain mild conditions that are sufficient to make multiagent planning amenable to a scalable approximation w. r. t. the number of agents. This is achieved by constructing a graphical model in which likelihood maximization is equivalent to plan optimization. Using the Expectation-Maximization framework for likelihood maximization, we show that the necessary inference can be decomposed into processes that often involve a small subset of agents, thereby facilitating scalability. We derive a global update rule that combines these local inferences to monotonically increase the overall solution quality. Experiments on a large multiagent planning benchmark confirm the benefits of the new approach in terms of runtime and scalability.

ICML Conference 2011 Conference Paper

Task Space Retrieval Using Inverse Feedback Control

• Nikolay Jetchev
• Marc Toussaint

NeurIPS Conference 2010 Conference Paper

An Approximate Inference Approach to Temporal Optimization in Optimal Control

• Konrad Rawlik
• Marc Toussaint
• Sethu Vijayakumar

Algorithms based on iterative local approximations present a practical approach to optimal control in robotic systems. However, they generally require the temporal parameters (for e. g. the movement duration or the time point of reaching an intermediate goal) to be specified \textit{a priori}. Here, we present a methodology that is capable of jointly optimising the temporal parameters in addition to the control command profiles. The presented approach is based on a Bayesian canonical time formulation of the optimal control problem, with the temporal mapping from canonical to real time parametrised by an additional control variable. An approximate EM algorithm is derived that efficiently optimises both the movement duration and control commands offering, for the first time, a practical approach to tackling generic via point problems in a systematic way under the optimal control framework. The proposed approach is evaluated on simulations of a redundant robotic plant.

ICRA Conference 2010 Conference Paper

Integrated motor control, planning, grasping and high-level reasoning in a blocks world using probabilistic inference

• Marc Toussaint
• Nils Plath
• Tobias Lang 0001
• Nikolay Jetchev

A new approach to planning and goal-directed behavior has recently been proposed using probabilistic inference in a graphical model that represents states, actions, constraints and goals of the future to infer appropriate actions and controls. The approach has led to new algorithms on the control and trajectory optimization level as well as for high-level rule-based planning in relational domains. In this paper we integrate these methods to a coherent control, trajectory optimization, and action planning architecture, using the principle of planning by inference across all levels of abstractions. Our scenario is a real blocks world: using a 14DoF Schunk arm and hand with tactile sensors and a stereo camera, the goal is to manipulate a set of objects on the table in a goal-oriented way. For high-level reasoning, we learn relational rule-based models from experience in simulation.

ICML Conference 2010 Conference Paper

Probabilistic Backward and Forward Reasoning in Stochastic Relational Worlds

• Tobias Lang 0001
• Marc Toussaint

ICRA Conference 2010 Conference Paper

Trajectory prediction in cluttered voxel environments

• Nikolay Jetchev
• Marc Toussaint

Trajectory planning and optimization is a fundamental problem in articulated robotics. It is often viewed as a two phase problem of initial feasible path planning around obstacles and subsequent optimization of a trajectory satisfying dynamical constraints. There are many methods that can generate good movements when given enough time, but planning for high-dimensional robot configuration spaces in realistic environments with many objects in real time remains challenging. This work presents a novel way for faster movement planning in such environments by predicting good path initializations. We build on our previous work on trajectory prediction by adapting it to environments modeled with voxel grids and defining a frame invariant prototype trajectory space. The constructed representations can generalize to a wide range of situations, allowing to predict good movement trajectories and speed up convergence of robot motion planning. An empirical comparison of the effect on planning movements with a combination of different trajectory initializations and local planners is presented and tested on a Schunk arm manipulation platform with laser sensors in simulation and hardware.

ICML Conference 2009 Conference Paper

Approximate inference for planning in stochastic relational worlds

• Tobias Lang 0001
• Marc Toussaint

ICML Conference 2009 Conference Paper

Model-free reinforcement learning as mixture learning

• Nikos Vlassis
• Marc Toussaint

ICML Conference 2009 Conference Paper

Multi-class image segmentation using conditional random fields and global classification

• Nils Plath
• Marc Toussaint
• Shinichi Nakajima

ICML Conference 2009 Conference Paper

Robot trajectory optimization using approximate inference

• Marc Toussaint

ICML Conference 2009 Conference Paper

Trajectory prediction: learning to map situations to robot trajectories

• Nikolay Jetchev
• Marc Toussaint

Trajectory planning and optimization is a fundamental problem in articulated robotics. Algorithms used typically for this problem compute optimal trajectories from scratch in a new situation. In effect, extensive data is accumulated containing situations together with the respective optimized trajectories - but this data is in practice hardly exploited. The aim of this paper is to learn from this data. Given a new situation we want to predict a suitable trajectory which only needs minor refinement by a conventional optimizer. Our approach has two essential ingredients. First, to generalize from previous situations to new ones we need an appropriate situation descriptor - we propose a sparse feature selection approach to find such well-generalizing features of situations. Second, the transfer of previously optimized trajectories to a new situation should not be made in joint angle space - we propose a more efficient task space transfer of old trajectories to new situations. Experiments on a simulated humanoid reaching problem show that we can predict reasonable motion prototypes in new situations for which the refinement is much faster than an optimization from scratch.

UAI Conference 2008 Conference Paper

Hierarchical POMDP Controller Optimization by Likelihood Maximization

• Marc Toussaint
• Laurent Charlin
• Pascal Poupart

Planning can often be simplified by decomposing the task into smaller tasks arranged hierarchically. Charlin et al. [4] recently showed that the hierarchy discovery problem can be framed as a non-convex optimization problem. However, the inherent computational difficulty of solving such an optimization problem makes it hard to scale to realworld problems. In another line of research, Toussaint et al. [18] developed a method to solve planning problems by maximumlikelihood estimation. In this paper, we show how the hierarchy discovery problem in partially observable domains can be tackled using a similar maximum likelihood approach. Our technique first transforms the problem into a dynamic Bayesian network through which a hierarchical structure can naturally be discovered while optimizing the policy. Experimental results demonstrate that this approach scales better than previous techniques based on non-convex optimization.

IROS Conference 2008 Conference Paper

Task maps in humanoid robot manipulation

• Michael Gienger
• Marc Toussaint
• Christian Goerick

This paper presents an integrative approach to solve the coupled problem of reaching and grasping an object in a cluttered environment with a humanoid robot. While finding an optimal grasp is often treated independently from reaching to the object, in most situations it depends on how the robot can reach a pregrasp pose while avoiding obstacles. We tackle this problem by introducing the concept of task maps which represent the manifold of feasible grasps for an object. Rather than defining a single end-effector goal position, a task map defines a goal hyper volume in the task space. We show how to efficiently learn such maps using the rapidly exploring random tree algorithm. Further, we generalise a previously developed motion optimisation scheme, based on a sequential attractor representation of motion, to cope with such task maps. The optimisation procedure incorporates the robotpsilas redundant whole body controller and uses analytic gradients to jointly optimise the motion costs (including criteria such as collision and joint limit avoidance, energy efficiency, etc.) and the choice of the grasp on the manifold of valid grasps. This leads to a preference of grasps which are easy to reach. The approach is demonstrated in two reach-grasp simulation scenarios with the humanoid robot ASIMO.

IJCAI Conference 2007 Conference Paper

• Ben H Williams
• Marc Toussaint
• Amos J Storkey

Biological movement control and planning is based upon motor primitives. In our approach, we presume that each motor primitive takes responsibility for controlling a small sub-block of motion, containing coherent muscle activation outputs. A central timing controller cues these subroutines of movement, creating complete movement strategies that are built up by overlaying primitives, thus creating synergies of muscle activation. This partitioning allows the movement to be defined by a sparse code representing the timing of primitive activations. This paper shows that it is possible to use a factorial hidden Markov model to infer primitives in handwriting data. The variation in the handwriting data can to a large extent be explained by timing variation in the triggering of the primitives. Once an appropriate set of primitives has been inferred, the characters can be represented as a set of timings of primitive activations, along with variances, giving a very compact representation of the character. The model is naturally partitioned into a low level primitive output stage, and a top-down primitive timing stage. This partitioning gives us an insight into behaviours such as scribbling, and what is learnt in order to write a new character.

NeurIPS Conference 2007 Conference Paper

Modelling motion primitives and their timing in biologically executed movements

• Ben Williams
• Marc Toussaint
• Amos Storkey

Biological movement is built up of sub-blocks or motion primitives. Such primitives provide a compact representation of movement which is also desirable in robotic control applications. We analyse handwriting data to gain a better understanding of use of primitives and their timings in biological movements. Inference of the shape and the timing of primitives can be done using a factorial HMM based model, allowing the handwriting to be represented in primitive timing space. This representation provides a distribution of spikes corresponding to the primitive activations, which can also be modelled using HMM architectures. We show how the coupling of the low level primitive model, and the higher level timing model during inference can produce good reconstructions of handwriting, with shared primitives for all characters modelled. This coupled model also captures the variance profile of the dataset which is accounted for by spike timing jitter. The timing code provides a compact representation of the movement while generating a movement without an explicit timing model produces a scribbling style of output.

IROS Conference 2007 Conference Paper

Probabilistic inference for structured planning in robotics

• Marc Toussaint
• Christian Goerick

Real-world robotic environments are highly structured. The scalability of planning and reasoning methods to cope with complex problems in such environments crucially depends on exploiting this structure. We propose a new approach to planning in robotics based on probabilistic inference. The method uses structured Dynamic Bayesian Networks to represent the scenario and efficient inference techniques (loopy belief propagation) to solve planning problems. In principle, any kind of factored or hierarchical state representations can be accounted for. We demonstrate the approach on reaching tasks under collision avoidance constraints with a humanoid upper body.

TCS Journal 2006 Journal Article

Compact representations as a search strategy: Compression EDAs

• Marc Toussaint

The choice of representation crucially determines the capability of search processes to find complex solutions in which many variables interact. The question is how good representations can be found and how they can be adapted online to account for what can be learned about the structure of the problem from previous samples. We address these questions in a scenario that we term indirect Estimation-of-Distribution: We consider a decorrelated search distribution (mutational variability) on a variable length genotype space. A one-to-one encoding onto the phenotype space then needs to induce an adapted phenotypic search distribution incorporating the dependencies between phenotypic variables that have been observed successful previously. Formalizing this in the framework of Estimation-of-Distribution Algorithms, an adapted phenotypic search distribution can be characterized as minimizing the Kullback–Leibler divergence (KLD) to a population of previously selected samples (parents). The paper derives a relation between this KLD and the description length of the encoding, stating that compact representations provide a way to minimize the divergence. A proposed class of Compression Evolutionary Algorithms and experiments with an grammar-based compression scheme illustrate the new concept.

ICML Conference 2006 Conference Paper

Probabilistic inference for solving discrete and continuous state Markov Decision Processes

• Marc Toussaint
• Amos J. Storkey

Inference in Markov Decision Processes has recently received interest as a means to infer goals of an observed action, policy recognition, and also as a tool to compute policies. A particularly interesting aspect of the approach is that any existing inference technique in DBNs now becomes available for answering behavioral question--including those on continuous, factorial, or hierarchical state representations. Here we present an Expectation Maximization algorithm for computing optimal policies. Unlike previous approaches we can show that this actually optimizes the discounted expected future return for arbitrary reward functions and without assuming an ad hoc finite total time. The algorithm is generic in that any inference technique can be utilized in the E-step. We demonstrate this for exact inference on a discrete maze and Gaussian belief state propagation in continuous stochastic optimal control problems.

IJCAI Conference 2005 Conference Paper

Learning discontinuities for switching between local models

• Marc Toussaint
• Sethu

ICML Conference 2005 Conference Paper

Learning discontinuities with products-of-sigmoids for switching between local models

• Marc Toussaint
• Sethu Vijayakumar

NeurIPS Conference 2003 Conference Paper

Learning a World Model and Planning with a Self-Organizing, Dynamic Neural System

• Marc Toussaint

We present a connectionist architecture that can learn a model of the relations between perceptions and actions and use this model for be- havior planning. State representations are learned with a growing self- organizing layer which is directly coupled to a perception and a motor layer. Knowledge about possible state transitions is encoded in the lat- eral connectivity. Motor signals modulate this lateral connectivity and a dynamic field on the layer organizes a planning process. All mecha- nisms are local and adaptation is based on Hebbian ideas. The model is continuous in the action, perception, and time domain.