Author name cluster

Tamim Asfour

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

107 papers

1 author row

ICLR Conference 2025 Conference Paper

A Riemannian Framework for Learning Reduced-order Lagrangian Dynamics

Katharina Friedl
Noémie Jaquier
Jens Lundell
Tamim Asfour
Danica Kragic

By incorporating physical consistency as inductive bias, deep neural networks display increased generalization capabilities and data efficiency in learning nonlinear dynamic models. However, the complexity of these models generally increases with the system dimensionality, requiring larger datasets, more complex deep networks, and significant computational effort. We propose a novel geometric network architecture to learn physically-consistent reduced-order dynamic parameters that accurately describe the original high-dimensional system behavior. This is achieved by building on recent advances in model-order reduction and by adopting a Riemannian perspective to jointly learn a non-linear structure-preserving latent space and the associated low-dimensional dynamics. Our approach enables accurate long-term predictions of the high-dimensional dynamics of rigid and deformable systems with increased data efficiency by inferring interpretable and physically-plausible reduced Lagrangian models.

ICRA Conference 2025 Conference Paper

Force Myography Based Torque Estimation in Human Knee and Ankle Joints

Charlotte Marquardt
Arne Schulz
Miha Dezman
Gunther Kurz
Thorsten Stein
Tamim Asfour

The online adaptation of exoskeleton control based on muscle activity sensing offers a promising approach to personalizing exoskeleton behavior based on the user's biosignals. While electromyography (EMG)-based methods have demonstrated improvements in joint torque estimation, EMG sensors require direct skin contact and extensive post-processing. In contrast, force myography (FMG) measures normal forces resulting from changes in muscle volume due to muscle activity. We propose an FMG-based method to estimate knee and ankle joint torques by integrating joint angles and velocities with muscle activity data. We learn a model for joint torque estimation using Gaussian process regression (GPR). The effectiveness of the proposed FMG-based method is validated on isokinetic motions performed by ten participants. The model is compared to a baseline model that uses only joint angle and velocity as well as a model augmented by EMG data. The results indicate that incorporating FMG into exoskeleton control can improve the estimation of joint torque for the ankle and knee joints in novel task characteristics within a single participant. Although the findings suggest that this approach may not improve the generalizability of estimates between multiple participants, they highlight the need for further research into its potential applications in exoskeleton control.

ICML Conference 2025 Conference Paper

Geometric Contact Flows: Contactomorphisms for Dynamics and Control

Andrea Testa
Søren Hauberg
Tamim Asfour
Leonel Rozo

Accurately modeling and predicting complex dynamical systems, particularly those involving force exchange and dissipation, is crucial for applications ranging from fluid dynamics to robotics, but presents significant challenges due to the intricate interplay of geometric constraints and energy transfer. This paper introduces Geometric Contact Flows (GFC), a novel framework leveraging Riemannian and Contact geometry as inductive biases to learn such systems. GCF constructs a latent contact Hamiltonian model encoding desirable properties like stability or energy conservation. An ensemble of contactomorphisms then adapts this model to the target dynamics while preserving these properties. This ensemble allows for uncertainty-aware geodesics that attract the system’s behavior toward the data support, enabling robust generalization and adaptation to unseen scenarios. Experiments on learning dynamics for physical systems and for controlling robots on interaction tasks demonstrate the effectiveness of our approach.

IROS Conference 2025 Conference Paper

The KIT Robotic Hands - A Scalable Humanoid Hand Platform With Multi-Modal Sensing and In-Hand Embedded Processing

Julia Starke
Felix Hundhausen
Pascal Weiner
Samuel Rader
Engjell Hyseni
Tamim Asfour

Humanoid robotic hands need to be versatile and capable of providing environmental information in order to serve as a platform for intelligent grasp control. To facilitate the design process of such hands, we present the KIT Robotic Hands. They have been designed to meet diverse application requirements through their scalability in size, actuation, sensorization and computing resources. The hands integrate a multi-modal sensor system, in-hand embedded processing capabilities, an adaptive underactuated mechanism and a continuously controllable thumb rotation to enhance dexterity. The flexibility of the design is demonstrated through two application-specific hand implementations: one is the ARMAR-7 hand, which has human hand dimensions for grasping daily objects in household tasks, the other is the ARMAR-DE hand, a larger hand designed for grasping bigger objects in decontamination tasks. We describe the design and mechatronics of the hands as well as an evaluation of the grasp success and image segmentation based on an in-hand integrated camera and onboard processing of visual data.

ICRA Conference 2025 Conference Paper

TWIN: Two-handed Intelligent Benchmark for Bimanual Manipulation

Markus Grotz
Mohit Shridhar
Yu-Wei Chao
Tamim Asfour
Dieter Fox

Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. While there exist several real-world bimanual systems, there is a lack of simulated benchmarks with a large task diversity for systematically studying bimanual capabilities across a wide range of tabletop tasks. This paper addresses the gap by presenting a benchmark for bimanual manipulation. A key functionality is the ability to autonomously generate training data without the necessity of human demonstrations to the robot. We open-source our code and benchmark, which comprises 13 new tasks with 23 unique task variations, each requiring a high degree of coordination and adaptability. To initiate the benchmark, we extended multiple state-of-the-art techniques to the domain of bimanual manipulation. The project website with code is available at: http://bimanual.github.io.

ICRA Conference 2024 Conference Paper

Ankle Exoskeleton with a Symmetric 3 DoF Structure for Plantarflexion Assistance

Miha Dezman
Charlotte Marquardt
Tamim Asfour

Ankle exoskeletons can assist the ankle joint and reduce the metabolic cost of walking. However, many existing ankle exoskeletons constrain the natural 3 degrees of freedom (DoF) of the ankle to limit the exoskeleton’s weight and mechanical complexity, thereby compromising comfort and kinematic compatibility with the user. This paper presents a novel ankle exoskeleton frame design that allows for 3 DoF ankle motion using a symmetric parallel frame design principle resulting in a strong frame while weighing 1. 8 kg. Furthermore, a cable routing method is proposed to actuate the plantarflexion of the ankle. The kinematic compatibility of the proposed exoskeleton frame is evaluated in straight- and curve-walking scenarios with four users. The study demonstrates that the exoskeleton frame adapts to the natural 3 DoF ankle motion and the range of motion (RoM) during walking. The actuation in plantarflexion is evaluated in a stationary torque experiment demonstrating the ability of the frame to transfer large torque loads of up to 57. 4 Nm. This work contributes to the design and development of more flexible and adaptable ankle exoskeletons for walking assistance.

IROS Conference 2024 Conference Paper

Beyond Feasibility: Efficiently Planning Robotic Assembly Sequences That Minimize Assembly Path Lengths

Alexander Cebulla
Tamim Asfour
Torsten Kröger

Advancements in Industry 4. 0 demand sophisticated solutions for automatic robotic assembly sequence planning (RASP), capable of handling the diversity and complexity of modern manufacturing tasks. One approach to RASP is Assembly-by-Disassembly (AbD). It first searches for a disassembly sequence that is then inverted to obtain an assembly sequence. One of the challenges of AbD, however, is the exponential number of potential assembly sequences for any given assembly. To mitigate this challenge, we propose to transfer knowledge obtained during previous planning attempts. Specifically, we present an approach that combines Monte Carlo Tree Search (MCTS) with deep Q-learning to optimize the total length of robotic assembly paths. We use a graph-based representation of disassembly states in combination with a graph neural network to learn the Q-function. We further discuss a principled approach to generate 3D assemblies out of aluminium profiles that a single robot manipulator can assemble. With this approach, we generated two datasets consisting of 14 assemblies with 21 removable parts and 7 assemblies with 30 removable parts. Using leave-one-out cross-validation, we were able to demonstrate how our approach outperformed an unmodified MCTS. Moreover, we successfully transferred knowledge between datasets.

ICRA Conference 2024 Conference Paper

Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks

Jianfeng Gao 0002
Xiaoshu Jin
Franziska Krebs
Noémie Jaquier
Tamim Asfour

Visual imitation learning has achieved impressive progress in learning unimanual manipulation tasks from a small set of visual observations, thanks to the latest advances in computer vision. However, learning bimanual coordination strategies and complex object relations from bimanual visual demonstrations, as well as generalizing them to categorical objects in novel cluttered scenes remain unsolved challenges. In this paper, we extend our previous work on keypoints-based visual imitation learning (K-VIL) [1] to bimanual manipulation tasks. The proposed Bi-KVIL jointly extracts so-called Hybrid Master-Slave Relationships (HMSR) among objects and hands, bimanual coordination strategies, and sub-symbolic task representations. Our bimanual task representation is object-centric, embodiment-independent, and viewpoint-invariant, thus generalizing well to categorical objects in novel scenes. We evaluate our approach in various real-world applications, showcasing its ability to learn fine-grained bimanual manipulation tasks from a small number of human demonstration videos. Videos and source code are available at https://sites.google.com/view/bi-kvil.

ICML Conference 2024 Conference Paper

Bringing Motion Taxonomies to Continuous Domains via GPLVM on Hyperbolic manifolds

Noémie Jaquier
Leonel Rozo
Miguel González Duque
Viacheslav Borovitskiy
Tamim Asfour

Human motion taxonomies serve as high-level hierarchical abstractions that classify how humans move and interact with their environment. They have proven useful to analyse grasps, manipulation skills, and whole-body support poses. Despite substantial efforts devoted to design their hierarchy and underlying categories, their use remains limited. This may be attributed to the lack of computational models that fill the gap between the discrete hierarchical structure of the taxonomy and the high-dimensional heterogeneous data associated to its categories. To overcome this problem, we propose to model taxonomy data via hyperbolic embeddings that capture the associated hierarchical structure. We achieve this by formulating a novel Gaussian process hyperbolic latent variable model that incorporates the taxonomy structure through graph-based priors on the latent space and distance-preserving back constraints. We validate our model on three different human motion taxonomies to learn hyperbolic embeddings that faithfully preserve the original graph structure. We show that our model properly encodes unseen data from existing or new taxonomy categories, and outperforms its Euclidean and VAE-based counterparts. Finally, through proof-of-concept experiments, we show that our model may be used to generate realistic trajectories between the learned embeddings.

ICRA Conference 2024 Conference Paper

Forgetting in Robotic Episodic Long-Term Memory

Joana Plewnia
Fabian Peller-Konrad
Tamim Asfour

Artificial cognitive architectures traditionally rely on complex memory models to encode, store, and retrieve information. However, the conventional practice of transferring all data from working memory (WM) to long-term memory (LTM) leads to high data volumes and challenges in efficient information processing and access. Deciding what information to retain or discard within a robot’s LTM is particularly challenging since knowledge about future data utilization is absent. Drawing inspiration from human forgetting this paper implements and evaluates novel forgetting techniques that allow consolidation in the robot’s LTM only when new information is encountered. The proposed approach combines fast filtering during data transfer to the robot’s LTM with slower yet more precise forgetting mechanisms that are periodically evaluated for offline data deletion inside the LTM. We compare different mechanisms, utilizing metrics such as data similarity, data age, and consolidation frequency. The efficacy of forgetting techniques is evaluated by comparing their performance in a task where two ARMAR robots search through their LTM for past object locations in episodic ego-centric images and robot state data. Experimental results show that our forgetting techniques significantly reduce the space requirements of a robot’s LTM while maintaining its capacity to successfully perform tasks relying on LTM information. Notably, similarity-based forgetting methods outperform frequency- and time-based approaches. The combination of online frequency-based, online similarity-based, offline similarity-based, and time-based decay methods shows superior performance compared to using individual forgetting strategies.

IROS Conference 2024 Conference Paper

Formalization of Temporal and Spatial Constraints of Bimanual Manipulation Categories

Franziska Krebs
Tamim Asfour

Executing bimanual manipulation tasks on humanoid robots introduces additional challenges due to inherent spatial and temporal coordination between both hands. In our previous work, we proposed the Bimanual Manipulation Taxonomy, which defines categories of bimanual manipulation strategies based on the coordination and physical interaction between both hands, the role of each hand in the task, and the symmetry of arm movements during task execution. In this work, we build upon this taxonomy and provide a formalization of temporal and spatial constraints associated with each category of the taxonomy. This formalization uses Petri nets to represent temporal constraints and differentiates between relative and global targets. We incorporate these constraints in a category-specific controller to enable reactive adaptation of the behavior according to the respective coordination constraints. We evaluated our approach in simulation and in real-world experiments on the humanoid robot ARMAR-6. The results demonstrate that category-specific constraints can be enforced when needed while maintaining flexibility to accommodate additional constraints.

ICRA Conference 2024 Conference Paper

Incremental Learning of Full-Pose Via-Point Movement Primitives on Riemannian Manifolds

Tilman Daab
Noémie Jaquier
Christian R. G. Dreher
André Meixner
Franziska Krebs
Tamim Asfour

Movement primitives (MPs) are compact representations of robot skills that can be learned from demonstrations and combined into complex behaviors. However, merely equipping robots with a fixed set of innate MPs is insufficient to deploy them in dynamic and unpredictable environments. Instead, the full potential of MPs remains to be attained via adaptable, large-scale MP libraries. In this paper, we propose a set of seven fundamental operations to incrementally learn, improve, and re-organize MP libraries. To showcase their applicability, we provide explicit formulations of the five spatial operations for libraries composed of Via-Point Movement Primitives (VMPs). By building on Riemannian manifold theory, our approach enables the incremental learning of all parameters of position and orientation VMPs within a library. Moreover, our approach stores a fixed number of parameters, thus complying with the essential principles of incremental learning. We evaluate our approach to incrementally learn a VMP library from sequentially-provided motion capture data.

ICRA Conference 2024 Conference Paper

Kinematic Synergy Primitives for Human-Like Grasp Motion Generation

Julia Starke
Tamim Asfour

Grasping with five-fingered humanoid hands is a complex control problem. Throughout the entire grasping motion, all finger joints need to be coordinated to achieve a stable grasp. Grasp synergies provide a simplified, low-dimensional representation of grasp postures and motions, that can be used for the description of human grasps as well as the generation of novel, human-like grasps. However, the abstract synergy representation complicates the association of relevant high-level grasp parameters, as for example the grasp type and final posture or the grasp speed. Therefore, it is difficult to control these grasp characteristics in the synergy space. This paper presents an adaptable representation for kinematic grasping motions in synergy space, that allows the generation of novel, human-like grasps under direct control of high-level grasp parameters. It is based on via-point movement primitives trained on synergy trajectories of human grasping motions. The representation using synergy primitives allows for a straightforward adaptation of grasp characteristics while preserving the essential grasping motion learned from human demonstration. The kinematic synergy primitives have a low reproduction error of 3. 9% of the maximum finger joint angle and are able to generate successful grasps on a simulated human hand and a real prosthetic hand.

IROS Conference 2024 Conference Paper

Learning Symbolic and Subsymbolic Temporal Task Constraints from Bimanual Human Demonstrations

Christian R. G. Dreher
Tamim Asfour

Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the subsymbolic level such as the duration of different actions, or their starting and end points in time. Such temporal constraints are crucial for temporal planning, reasoning, and the exact timing for the execution of bimanual actions on a bimanual robot. In our previous work, we addressed the learning of temporal task constraints on the symbolic level and demonstrated how a robot can leverage this knowledge to respond to failures during execution. In this work, we propose a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations. Our main contributions are a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints, as well as a method based on fuzzy logic to derive symbolic temporal task constraints from this representation. This complements our previous work on learning comprehensive temporal task models by integrating symbolic and subsymbolic information based on a subsymbolic foundation, while still maintaining the symbolic expressiveness of our previous approach. We compare our proposed approach with our previous pure-symbolic approach and show that we can reproduce and even outperform it. Additionally, we show how the subsymbolic temporal task constraints can synchronize otherwise unimanual movement primitives for bimanual behavior on a humanoid robot.

IROS Conference 2024 Conference Paper

MAkEable: Memory-centered and Affordance-based Task Execution Framework for Transferable Mobile Manipulation Skills

Christoph Pohl
Fabian Reister
Fabian Peller-Konrad
Tamim Asfour

To perform versatile mobile manipulation tasks in human-centered environments, the ability to efficiently transfer learned skills, knowledge, and experiences from one robot to another or across different environments is critical. In this paper, we present MAkEable, a versatile uni- and multi-manual mobile manipulation framework that facilitates the transfer of capabilities and knowledge across different tasks, environments, and robots. Our framework integrates an affordance-based task description into the memory-centric cognitive architecture of the ARMAR humanoid robot family, which supports the sharing of experiences and demonstrations for transferring mobile manipulation skills. By representing mobile manipulation actions through affordances, i. e. , interaction possibilities of the robot with its environment, we provide a unifying framework for the autonomous uni- and multi-manual manipulation of known and unknown objects in various environments. We demonstrate MAkEable’s applicability in real-world experiments for multiple robots, tasks, and environments. This includes grasping known and unknown objects, object placing, bimanual object grasping, memory-enabled skill transfer in a drawer opening scenario across two different humanoid robots, and a pouring task learned from human demonstration. Code is available through our project page 1.

IROS Conference 2024 Conference Paper

Riemannian Flow Matching Policy for Robot Motion Learning

Max Braun
Noémie Jaquier
Leonel Rozo
Tamim Asfour

We introduce Riemannian Flow Matching Policies (RFMP), a novel model for learning and synthesizing robot visuomotor policies. RFMP leverages the efficient training and inference capabilities of flow matching methods. By design, RFMP inherits the strengths of flow matching: the ability to encode high-dimensional multimodal distributions, commonly encountered in robotic tasks, and a very simple and fast inference process. We demonstrate the applicability of RFMP to both state-based and vision-conditioned robot motion policies. Notably, as the robot state resides on a Riemannian manifold, RFMP inherently incorporates geometric awareness, which is crucial for realistic robotic tasks. To evaluate RFMP, we conduct two proof-of-concept experiments, comparing its performance against Diffusion Policies. Although both approaches successfully learn the considered tasks, our results show that RFMP provides smoother action trajectories with significantly lower inference times.

ICRA Conference 2024 Conference Paper

Towards Unifying Human Likeness: Evaluating Metrics for Human-Like Motion Retargeting on Bimanual Manipulation Tasks

André Meixner
Mischa Carl
Franziska Krebs
Noémie Jaquier
Tamim Asfour

Generating human-like robot motions is pivotal for achieving smooth human-robot interactions. Such motions contribute to better predictions of robot motions by humans, thus leading to more intuitive interaction and increased acceptability. Human likeness in robot motions has been conventionally measured and realized via the optimization of human-likeness metrics. However, the abundance of such metrics and the absence of standardized criteria impede their usage in novel contexts. In this work, we introduce a unified human-likeness metric built from a hierarchically weighted sum of individual metrics. The proposed metric is derived from a thorough analysis of eleven existing human-likeness criteria and is applicable across various tasks and robot models. We evaluate its performance in the context of motion retargeting of bimanual tasks with three different humanoid robots.

ICRA Conference 2024 Conference Paper

Unraveling the Single Tangent Space Fallacy: An Analysis and Clarification for Applying Riemannian Geometry in Robot Learning

Noémie Jaquier
Leonel Rozo
Tamim Asfour

In the realm of robotics, numerous downstream robotics tasks leverage machine learning methods for processing, modeling, or synthesizing data. Often, this data comprises variables that inherently carry geometric constraints, such as the unit-norm condition of quaternions representing rigid-body orientations or the positive definiteness of stiffness and manipulability ellipsoids. Handling such geometric constraints effectively requires the incorporation of tools from differential geometry into the formulation of machine learning methods. In this context, Riemannian manifolds emerge as a powerful mathematical framework to handle such geometric constraints. Nevertheless, their recent adoption in robot learning has been largely characterized by a mathematically-flawed simplification, hereinafter referred to as the "single tangent space fallacy". This approach involves merely projecting the data of interest onto a single tangent (Euclidean) space, over which an off-the-shelf learning algorithm is applied. This paper provides a theoretical elucidation of various misconceptions surrounding this approach and offers experimental evidence of its shortcomings. Finally, it presents valuable insights to promote best practices when employing Riemannian geometry within robot learning applications.

IROS Conference 2024 Conference Paper

Visual Imitation Learning of Task-Oriented Object Grasping and Rearrangement

Yichen Cai 0007
Jianfeng Gao 0002
Christoph Pohl
Tamim Asfour

Task-oriented object grasping and rearrangement are key skills for robots, which have to perform versatile real-world manipulation tasks. However, they remain challenging due to partial observations of the objects and shape variations in categorical objects. In this paper, we present the Multi-feature Implicit Model (MIMO), a novel object representation that encodes multiple spatial features between a point and an object in an implicit neural field. Training such a model on multiple features ensures that it embeds the object shapes consistently in different aspects, thus improving its performance in object shape reconstruction from partial observation, shape similarity measure, and modeling spatial relations between objects. Based on MIMO, we propose a framework to learn task-oriented object grasping and rearrangement from single or multiple human demonstration videos. The evaluations in simulation show that our approach outperforms the state-of-the-art methods for multi- and single-view observations. Real-world experiments demonstrate the efficacy of our approach in one- and few-shot imitation learning of manipulation tasks.

IROS Conference 2023 Conference Paper

An Evaluation of Action Segmentation Algorithms on Bimanual Manipulation Datasets

André Meixner
Franziska Krebs
Noémie Jaquier
Tamim Asfour

Humans naturally execute many everyday manipulation actions with both arms simultaneously. Similarly, endowing robots with bimanual manipulation task models is key to efficiently perform complex manipulation tasks. To do so, a promising approach is to learn a library of task models from human demonstrations. However, this requires human motions to be meaningfully segmented. In this paper, we propose to segment the motion of each hand individually to account for different bimanual coordination patterns and provide a thorough evaluation of state-of-the-art segmentation algorithms on bimanual manipulation datasets. In particular, we compare segmentation algorithms at trajectory and semantic level with hierarchical algorithms. Moreover, our evaluation extensively studies the performances of various segmentation algorithms over a novel extension of the KIT Bimanual Manipulation Dataset featuring ~ 176 minutes of human motion recordings in household scenarios.

IROS Conference 2023 Conference Paper

Combining Measurement Uncertainties with the Probabilistic Robustness for Safety Evaluation of Robot Systems

Woo-Jeong Baek
Christoph Ledermann
Tamim Asfour
Torsten Kröger

In this paper, we present a method to engage measurement uncertainties with the probabilistic robustness to one system uncertainty measure. Providing a metric indicating the potential occurrence of dangerous situations is highly essential for safety-critical robot applications. Due to the difficulty of finding a quantifiable, unambiguous representation however, such a metric has not been derived to date. In case of sensory devices, measurement uncertainties are usually provided by manufacturer specifications. Apart from that, several contributions demonstrate that the accuracy of neural networks is verifiable via the robustness. However, state-of-the-art literature is mainly concerned with theoretical investigations such that scarce attention has been devoted to the transfer of the robustness to real-world applications. To fill this gap, we show how the probabilistic robustness can be made useful for evaluating quantitative safety limits. Our key idea is to exploit the analogy between measurement uncertainties and the probabilistic robustness: While measurement uncertainties reflect possible shifts due to technical limitations, the robustness refers to the tolerated amount of distortions in the input data for an unaltered output. Inspired by this analogy, we combine both measures to quantify the system uncertainty online. We validate our method in different settings under real-world conditions. Our findings exemplify that incorporating the novel uncertainty metric effectively prevents the rate of dangerous situations in Human-Robot Collaboration.

IROS Conference 2023 Conference Paper

On the Design of Region-Avoiding Metrics for Collision-Safe Motion Generation on Riemannian Manifolds

Holger Klein
Noémie Jaquier
André Meixner
Tamim Asfour

The generation of energy-efficient and dynamic-aware robot motions that satisfy constraints such as joint limits, self-collisions, and collisions with the environment remains a challenge. In this context, Riemannian geometry offers promising solutions by identifying robot motions with geodesics on the so-called configuration space manifold. While this manifold naturally considers the intrinsic robot dynamics, constraints such as joint limits, self-collisions, and collisions with the environment remain overlooked. In this paper, we propose a modification of the Riemannian metric of the configuration space manifold allowing for the generation of robot motions as geodesics that efficiently avoid given regions. We introduce a class of Riemannian metrics based on barrier functions that guarantee strict region avoidance by systematically generating accelerations away from no-go regions in joint and task space. We evaluate the proposed Riemannian metric to generate energy-efficient, dynamic-aware, and collision-free motions of a humanoid robot as geodesics and sequences thereof.

ICRA Conference 2023 Conference Paper

Speeding Up Assembly Sequence Planning Through Learning Removability Probabilities

Alexander Cebulla
Tamim Asfour
Torsten Kröger

Industry 4. 0 facilitates a high number of product variants, posing significant challenges for modern manufacturing. One of them is the automatic creation of assembly sequences. This can be achieved with the assembly-by-disassembly (AbD) approach, which is currently highly inefficient. We aim at speeding up AbD by leveraging deep learning. AbD relies on iteratively testing parts for removal, which makes the order in which parts are tested highly relevant for its run-time. We optimize this order by training a graph neural network (GNN) based on the shape of parts and the shape of local part connections. For each part, it predicts a removability probability. We use these probabilities to optimize the order in which parts are tested for removal. This reduces the number of parts tested by approximately 64%-90%, depending on the tested product. Further improvements are achieved by combining our approach with bookkeeping, another approach for speeding up AbD. Finally, we separately analyze the impact of the parts and their connections on the removability probabilities predicted by the GNN. We found that most of the important information regarding a part's removability can be derived from its connections alone.

IROS Conference 2023 Conference Paper

Upper Bounds for Localization Errors in 2D Human Pose Estimation

Patrick Schlosser
Christoph Ledermann
Tamim Asfour

Obtaining reliable detections of a human is crucial for many safety-related robotic tasks. This can be done by human pose estimation methods, which predict the position of several different keypoints of the human body. In most cases, recent approaches based on neural networks produce ‘good’ results, i. e. predictions with small localization errors, however, large errors do also occur. For an individual keypoint prediction, the magnitude of the error is unknown, posing a risk to safety. In this work, we extend a neural network architecture for single-person 2D human pose estimation, so that it predicts not only the keypoints of the human body, but also corresponding upper bounds for their localization errors. These upper bounds correspond to the neural network's confidence in its output, and are obtained by one of two general strategies based on (i) a direct estimation of the localization error or (ii) the predicted standard deviations of a 2D Gaussian. We propose several approaches employing these strategies and evaluate them on the MPII Human Pose dataset. In addition, we consider two quality criteria for the results: closeness of the predicted keypoint position to the actual one, and closeness of the predicted upper bound to the localization error. The best results are achieved by a Gaussian-based approach, which predicted correct upper bounds in 94. 7% of the cases, while also sufficiently fulfilling the quality criteria.

IROS Conference 2022 Conference Paper

A Compact, Lightweight and Singularity-Free Wrist Joint Mechanism for Humanoid Robots

Cornelius Klas
Tamim Asfour

Building humanoid robots with properties similar to those of humans in terms of strength and agility is a great and unsolved challenge. This work introduces a compact and lightweight wrist joint mechanism that is singularity-free and has large range of motion. The mechanism provides two degrees of freedom (DoF) and was developed for integration into a human scale humanoid robot arm. It is based on a parallel mechanism with rolling contact joint behaviour and remote actuation that facilitates a compact design with low mass and inertia. The mechanism's kinematics along with a solution of the inverse kinematics problem for the specific design, and the manipulability analysis are presented. The first prototype of the proposed mechanism shows the possible integration of actuation, sensing and electronics in small and narrow space. Experimental evaluations shows that the design feature unique performance regarding weight, speed, payload and accuracy.

IROS Conference 2022 Conference Paper

A Riemannian Take on Human Motion Analysis and Retargeting

Holger Klein
Noémie Jaquier
André Meixner
Tamim Asfour

Dynamic motions of humans and robots are widely driven by posture-dependent nonlinear interactions between their degrees of freedom. However, these dynamical effects remain mostly overlooked when studying the mechanisms of human movement generation. Inspired by recent works, we hypothesize that human motions are planned as sequences of geodesic synergies, and thus correspond to coordinated joint movements achieved with piecewise minimum energy. The underlying computational model is built on Riemannian geometry to account for the inertial characteristics of the body. Through the analysis of various human arm motions, we find that our model segments motions into geodesic synergies, and successfully predicts observed arm postures, hand trajectories, as well as their respective velocity profiles. Moreover, we show that our analysis can further be exploited to transfer arm motions to robots by reproducing individual human synergies as geodesic paths in the robot configuration space.

IROS Conference 2022 Conference Paper

Learning Symbolic Failure Detection for Grasping and Mobile Manipulation Tasks

Patrick Hegemann
Tim Zechmeister
Markus Grotz
Kevin Hitzler
Tamim Asfour

The ability to detect failure during task execution and to recover from failure is vital for autonomous robots performing tasks in previously unknown environments. In this paper, we present an approach for failure detection during the execution of grasping and mobile manipulation tasks by a humanoid robot. The approach combines multi-modal sensory information consisting of proprioceptive, force and visual information to learn task models from multiple successful task executions, in order to detect failures and to externalize them for humans in an interpretable way. To this end, we define symbolic action predicates based on multi-modal sensory information to allow high-level state estimation based on action-specific decision trees. To allow symbolic failure detection, we then learn task models that are represented as Markov chains. We evaluated the approach in several pick-and-place and mobile manipulation tasks performed by a humanoid robot in a decommissioning and a household scenario. The evaluation shows that the learned task models are capable of detecting failure with an F1-score of 93 %.

IROS Conference 2022 Conference Paper

Learning Temporal Task Models from Human Bimanual Demonstrations

Christian R. G. Dreher
Tamim Asfour

Learning temporal relations between actions in a bimanual manipulation task is important for capturing the constraints of actions required to achieve the task's goal. However, given several demonstrations of a bimanual manipulation task, the problem of identifying the true temporal dependencies between actions - if there are any - is very challenging due to contradictions. We propose a model-driven approach for learning temporal task models from multiple bimanual human demonstrations that represents temporal relations on two levels. First, temporal relations between sets of actions that exhibit a tight temporal coupling, and second, temporal relations between these sets of actions. We build on Allen's interval algebra as a representation to express relations between temporal intervals. Semantically defining these interval relations allows us to soften their formulation to deal with inaccuracies in real data obtained when observing humans demonstrating the task. Our temporal task models can be learned incrementally from multiple modalities, and allow us to reason about viable alternatives during task execution in case of unexpected events. We evaluated the approach quantitatively on two datasets and qualitatively on a humanoid robot. The evaluation shows how inherent properties of bimanual human manipulation tasks can be exploited to derive a model useful for the reproduction by humanoid robots.

ICRA Conference 2022 Conference Paper

Oriented Surface Reachability Maps for Robot Placement

Timo Birr
Christoph Pohl
Tamim Asfour

For a robot to perform a grasping and manipulation task, it has to determine possible robot placements in the workspace, from which target objects or environmental elements relevant to the given task are reachable. This work presents a novel approach for finding placements for the mobile base of a humanoid robot in an unknown environment with multiple support planes. We propose a novel type of reachability map - the Oriented Surface Reachability Map - that takes inclined surfaces in the environment into account and has the same complexity as reachability maps designed for flat surfaces. The resulting robot placements are not limited to SE(2) but can be applied to arbitrarily oriented planes in 3D space. The proposed method was evaluated in simulation and on the humanoid robot ARMAR-6 in real-world grasping experiments. The results show that a placement can be found for over 80% of the poses that are reachable in complicated, simulated environments, with only a small runtime overhead.

IROS Conference 2022 Conference Paper

SpeedFolding: Learning Efficient Bimanual Folding of Garments

Yahav Avigal
Lars Berscheid
Tamim Asfour
Torsten Kröger
Ken Goldberg

Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this work, we develop SpeedFolding, a reliable and efficient bimanual system, which given user-defined instructions as folding lines, manipulates an initially crumpled garment to (1) a smoothed and (2) a folded configuration. Our primary contribution is a novel neural network architecture that is able to predict pairs of gripper poses to parameterize a diverse set of bimanual action primitives. After learning from 4300 human- annotated and self-supervised actions, the robot is able to fold garments from a random initial configuration in under 120 s on average with a success rate of 93 %. Real-world experiments show that the system is able to generalize to unseen garments of different color, shape, and stiffness. While prior work achieved 3–6 Folds Per Hour (FPH), SpeedFolding achieves 30–40 FPH. See https://pantor.github.io/speedfolding for code, videos, and datasets.

ICRA Conference 2021 Conference Paper

Binary-LoRAX: Low-Latency Runtime Adaptable XNOR Classifier for Semi-Autonomous Grasping with Prosthetic Hands

Nael Fasfous
Manoj Rohit Vemparala
Alexander Frickenstein
Mohamed Badawy
Felix Hundhausen
Julian Höfer
Naveen Shankar Nagaraja
Christian Unger

Intelligent, semi-autonomous prostheses take ad-vantage of combining autonomous functions and traditional myoelectric control. With the help of visual and environment sensors, intelligent prostheses achieve a level of autonomy which relieves the user from generating elaborate electromyographic (EMG) signals for grasp type and trajectory. To achieve the desired functionality, the semi-autonomous prosthesis must efficiently process the incoming environmental data at a high rate, with low power and high accuracy. In this paper, we propose Binary-LoRAX, a low-latency runtime adaptable classifier for the semi-autonomous grasping task of prosthetic hands. We offload the classification task to an efficient binary neural network accelerator which performs high-throughput XNOR operations on digital signal processing (DSP) blocks. To tailor the classifier’s performance to the current application scenario, we propose a frequency scaling approach which dynamically switches between two modes of operation, high-performance and power-saving. At high-performance, classifications are performed with a low latency of 0. 45ms, high-throughput of 4999 FPS and power consumption of ∼ 2. 15 W. This enables functions such as object localization and batch classification. Switching to power-saving mode, a latency of 80 ms is maintained, with up to 19% improved classifier battery-life. Our prototypes achieve a high accuracy of up to 99. 82% on a 25 class problem from the YCB graspable object dataset.

IROS Conference 2021 Conference Paper

Detecting Grasp Phases and Adaption of Object-Hand Interaction Forces of a Soft Humanoid Hand Based on Tactile Feedback

Pascal Weiner
Felix Hundhausen
Raphael Grimm
Tamim Asfour

Engineering humanoid robot hands with the ability to dexterously grasp objects of different sizes, shapes, mate-rial properties and weights requires sophisticated tactile sensing and intelligent controllers able to interpret sensory information and adapt contact forces with the object to achieve a stable and safe grasp. In this paper, we present a new soft humanoid hand equipped with a multimodal sensor system in each finger and a human-inspired grasp-phases controller that is able to detect the different phases of a grasping and manipulation task, adapt interaction forces with the manipulated object and balance the force distribution in both precision and power grasps based on tactile feedback. To evaluate the controller, we conducted experiments with the hand on the humanoid robot ARMAR-6 and 31 different soft and rigid everyday objects and food items with weights ranging from 4. 8 g of a paper cup to 1133. 8 g of a bottle, different shapes and material properties. The results show that grasping force can be reduced by 65% compared to a naive grasping approach using maximum force for grasping and manipulating both fragile objects without destruction as well as heavy objects.

IROS Conference 2021 Conference Paper

Fast Reactive Grasping with In-Finger Vision and In-Hand FPGA-accelerated CNNs

Felix Hundhausen
Raphael Grimm
Leon Stieber
Tamim Asfour

We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding high bandwidth raw data streaming via the robots real-time data bus. The hardware acceleration allows fast detection and localization of objects based on finger-camera images and provides input for a grasping controller. To this end, we implement a resource-aware encoder-decoder Convolutional Neural Network (CNN) for pixel-wise object segmentation and run inference on the in-hand embedded system at 3. 58 GOPS. We evaluate the system, consisting of the soft hand with in-finger vision and the in-hand FPGA-accelerated CNN in several experiments on the humanoid robot ARMAR-6. Specifically, we evaluate the overall system response time, the ability to perform precision grasps and test reactivity and reliability that are required for handover actions. We obtain an overall system response time of 154 ms for catching a falling object and obtain a success rate of 90 % reliability for the power drill handover tasks. Further, we successfully demonstrate ability of dexterous grasping and manipulation of a pencil from a cup.

IROS Conference 2021 Conference Paper

Graph-based Task-specific Prediction Models for Interactions between Deformable and Rigid Objects

Zehang Weng
Fabian Paus
Anastasiia Varava
Hang Yin 0001
Tamim Asfour
Danica Kragic

Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipulation, involving interactions between rigid objects and a deformable bag. The dataset incorporates a rich variety of scenarios including different object sizes, object numbers and manipulation actions. We approach dynamics learning by proposing an object-centric graph representation and two modules which are Active Prediction Module (APM) and Position Prediction Module (PPM) based on graph neural networks with an encode-process-decode architecture. At the inference stage, we build a two-stage model based on the learned modules for single time step prediction. We combine modules with different prediction horizons into a mixed-horizon model which addresses long-term prediction. In an ablation study, we show the benefits of the two-stage model for single time step prediction and the effectiveness of the mixed-horizon model for long-term prediction tasks. Supplementary material is available at https://github.com/wengzehang/deformable_rigid_interaction_prediction

IROS Conference 2021 Conference Paper

Temporal Force Synergies in Human Grasping

Julia Starke
Marco Keller
Tamim Asfour

Humans can intuitively grasp objects of different shape and weight. Throughout the grasp execution they control and coordinate the grasp forces at all contact points between the hand and the object to achieve a stable grasp. Dexterous grasping with humanoid hands relies on the perfect coordination between grasp posture and force balance at the contact points in a high dimensional space and remains a challenge. In this paper, we present temporal force synergies describing the change in human grasp forces during the grasp execution in a low-dimensional space based on two new grasp synergy models: 1) static force synergies that are derived by a Principal Component Analysis and represent temporal grasp forces as a sequence of time-independent synergy configurations and 2) dynamic force synergies that are learned by a recurrent neural network and encode the temporal change of grasp forces throughout grasp execution in a latent synergy space clustered by grasp types. We show that both synergy spaces encode human grasp forces with an error of less than 2% and allow the generation of human-like grasp force patterns. Grasp forces for stable grasps described by the dynamic force synergies achieve a grasp quality comparable to demonstrated human grasps in simulation.

ICRA Conference 2021 Conference Paper

The KIT Gripper: A Multi-Functional Gripper for Disassembly Tasks

Cornelius Klas
Felix Hundhausen
Jianfeng Gao 0002
Christian R. G. Dreher
Stefan Reither
You Zhou 0007
Tamim Asfour

We introduce a multi-functional robotic gripper equipped with a set of actions required for disassembly of electromechanical devices. The gripper consists of a robot arm with 5 degrees of freedom (DoF) for manipulation and a jaw gripper with a 1-DoF rotation joint and a 1-DoF closing joint. The system enables manipulation in 7 DoF and offers the ability to reposition objects in hand and to perform tasks that usually require bimanual systems. The sensor system of the gripper includes relative and absolute joint encoders, force and pressure sensors to provide feedback about interaction forces, a tool- mounted camera for screw detection and precise placement of the tool tip using image-based visual servoing. We present a data-driven method for estimating joint torques based on the output voltage and motor speed. Further, we provide methods for teaching disassembly actions based on human demonstration, their representation as movement primitives and execution based on sensory feedback. We provide quantitative results regarding positioning and torque estimation accuracy, disassembly success rate and qualitative results regarding the successful disassembly of hard disc drives.

ICRA Conference 2021 Conference Paper

Vision-Based Robotic Pushing and Grasping for Stone Sample Collection under Computing Resource Constraints

Raphael Grimm
Markus Grotz
Simon Ottenhaus
Tamim Asfour

Increasing the robustness of grasping actions and the recovery from failure is key to improving a robot’s autonomy. Endowing robots with the ability to robustly grasp and manipulate unknown difficult objects such as stones is required for sample collection in unknown environments. In this paper, we present a complete system for robust grasping of stones, which integrates stone segmentation based on depth information, the generation of grasp hypotheses and pushing actions as well as their execution. In particular, our system has been designed to solve these tasks on robots with limited computing resources. We evaluate the performance in real robot experiments in the context of stone sample collection. The results show that such a challenging task is achievable under computing resource constraints.

IROS Conference 2020 Conference Paper

A Soft Humanoid Hand with In-Finger Visual Perception

Felix Hundhausen
Julia Starke
Tamim Asfour

We present a novel underactued humanoid five finger soft hand, the KIT Finger-Vision Soft Hand, which is equipped with cameras in the fingertips and integrates a high performance embedded system for visual processing and control. We describe the actuation mechanism of the hand and the tendon-driven soft finger design with internally routed high-bandwidth flat-flex cables. For efficient on-board parallel processing of visual data from the cameras in each fingertip, we present a hybrid embedded architecture consisting of a field programmable logic array (FPGA) and a microcontroller that allows the realization of visual object segmentation based on convolutional neural networks. We evaluate the hand design by conducting durability experiments with one finger and quantify the grasp performance in terms of grasping force, speed and grasp success. The results show that the hand exhibits a grasp force of 31. 8 ± 1. 2 N and a mechanical durability of the finger of more than 15. 000 closing cycles. Finally, we evaluate the accuracy of visual object segmentation during the different phases of the grasping process using five different objects. Hereby, an accuracy above 90% can be achieved.

IROS Conference 2020 Conference Paper

Affordance-Based Grasping and Manipulation in Real World Applications

Christoph Pohl
Kevin Hitzler
Raphael Grimm
Antonio Zea 0001
Uwe D. Hanebeck
Tamim Asfour

In real world applications, robotic solutions remain impractical due to the challenges that arise in unknown and unstructured environments. To perform complex manipulation tasks in complex and cluttered situations, robots need to be able to identify the interaction possibilities with the scene, i. e. the affordances of the objects encountered. In unstructured environments with noisy perception, insufficient scene understanding and limited prior knowledge, this is a challenging task. In this work, we present an approach for grasping unknown objects in cluttered scenes with a humanoid robot in the context of a nuclear decommissioning task. Our approach combines the convenience and reliability of autonomous robot control with the precision and adaptability of teleoperation in a semi-autonomous selection of grasp affordances. Additionally, this allows exploiting the expert knowledge of an experienced human worker. To evaluate our approach, we conducted 75 real world experiments with more than 660 grasp executions on the humanoid robot ARMAR-6. The results demonstrate that high-level decisions made by the human operator, supported by autonomous robot control, contribute significantly to successful task execution.

ICRA Conference 2020 Conference Paper

Predicting Pushing Action Effects on Spatial Object Relations by Learning Internal Prediction Models

Fabian Paus
Teng Huang
Tamim Asfour

Understanding the effects of actions is essential for planning and executing robot tasks. By imagining possible action consequences, a robot can choose specific action parameters to achieve desired goal states. We present an approach for parametrizing pushing actions based on learning internal prediction models. These pushing actions must fulfill constraints given by a high-level planner, e. g. , after the push the brown box must be to the right of the orange box. In this work, we represent the perceived scenes as object-centric graphs and learn an internal model, which predicts object pose changes due to pushing actions. We train this internal model on a large synthetic data set, which was generated in simulation, and record a smaller data set on the real robot for evaluation. For a given scene and goal state, the robot generates a set of possible pushing action candidates by sampling the parameter space and then evaluating the candidates by internal simulation, i. e. , by comparing the predicted effect resulting from the internal model with the desired effect provided by the high-level planner. In the evaluation, we show that our model achieves high prediction accuracy in scenes with a varying number of objects and, in contrast to state-of-the-art approaches, is able to generalize to scenes with more objects than seen during training. In experiments on the humanoid robot ARMAR-6, we validate the transfer from simulation and show that the learned internal model can be used to manipulate scenes into desired states effectively.

IROS Conference 2020 Conference Paper

Representing Spatial Object Relations as Parametric Polar Distribution for Scene Manipulation Based on Verbal Commands

Rainer Kartmann
You Zhou 0007
Danqing Liu
Fabian Paus
Tamim Asfour

Understanding spatial relations is a key element for natural human-robot interaction. Especially, a robot must be able to manipulate a given scene according to a human verbal command specifying desired spatial relations between objects. To endow robots with this ability, a suitable representation of spatial relations is necessary, which should be derivable from human demonstrations. We claim that polar coordinates can capture the underlying structure of spatial relations better than Cartesian coordinates and propose a parametric probability distribution defined in polar coordinates to represent spatial relations. We consider static spatial relations such as left of, behind, and near, as well as dynamic ones such as closer to and other side of, and take into account verbal modifiers such as roughly and a lot. We show that adequate distributions can be derived for various combinations of spatial relations and modifiers in a sample-efficient way using Maximum Likelihood Estimation, evaluate the effects of modifiers on the distribution parameters, and demonstrate our representation's usefulness in a pick-and-place task on a real robot.

IROS Conference 2019 Conference Paper

Learning Via-Point Movement Primitives with Inter- and Extrapolation Capabilities

You Zhou 0007
Jianfeng Gao 0002
Tamim Asfour

Movement Primitives (MPs) are a promising way for representing robot motions in a flexible and adaptable manner. Due to the simple and compact form, they have been widely used in robotics. A major goal of the research activities on MPs is to learn models, which can adapt to changing task constraints, e. g. new motion targets. However, the adaptability of current MPs is limited to a small set of constraints due to their simple structures. It is indeed not a trivial task to maintain the simplicity of MPs representation and, at the same time, enhance their adaptability. In this paper, we discuss the adaptability of popular MPs such as Dynamic Movement Primitives (DMP) and Probabilistic Movement Primitives (ProMP) and propose a new simple but efficient formulation of MPs, the Via-points Movement Primitive (VMP), that can adapt to arbitrary via-points using a simple structured model that is based on the previous approaches but outperforms those in terms of extrapolation abilities.

IROS Conference 2019 Conference Paper

Minimal Sensor Setup in Lower Limb Exoskeletons for Motion Classification based on Multi-Modal Sensor Data

Isabel Patzer
Tamim Asfour

Exoskeletons are considered to be a promising technology for assisting and augmenting human performance. A number of challenges related to design, intuitive control and interfaces to the human body must be addressed. In this paper, we approach the question of a minimal sensor setup for the realization of control strategies which take into account the actions currently performed by the user. To this end, we extend our previous work on online classifications of a human wearing a lower limb exoskeleton in two directions. First, we investigate the minimal number of sensors that should be attached to the exoskeleton to achieve a certain classification accuracy by investigating different sensor setups. We compare results of motion classification of 14 different daily activities such as walking forward and going upstairs using Hidden Markov Models. Second, we analyse the influence of different window sizes, as well as the classification performance of different motion types when training on multi- and single-subjects. Our results reveal that we can reduce our sensor setup significantly while achieving about the same classification performance.

IROS Conference 2019 Conference Paper

Predicting Grasp Success with a Soft Sensing Skin and Shape-Memory Actuated Gripper

Julian Zimmer
Tess Lee Hellebrekers
Tamim Asfour
Carmel Majidi
Oliver Kroemer

Tactile sensors have been increasingly used to support rigid robot grippers in object grasping and manipulation. However, rigid grippers are often limited in their ability to handle compliant, delicate, or irregularly shaped objects. In recent years, grippers made from soft and flexible materials have become increasingly popular for certain manipulation tasks, e. g. , grasping, due to their ability to conform to the object shape without the need for precise control. Although promising, such soft robot grippers currently suffer from the lack of available sensing modalities. In this work, we introduce a soft and stretchable sensing skin and incorporate it into the two fingers of a shape-memory actuated soft gripper. The onboard sensing skin includes a 9-axis inertial measurement unit (IMU) and five discrete pressure sensors per finger. We use this sensorized soft gripper to study grasp success and stability of over 2585 grasps with various objects using several machine learning methods. Our experiments show that LSTMs were the most accurate predictors of grasp success and stability, compared to SVMs, FFNNs, and ST-HMP. We also evaluated the effects on performance of each sensor's data, and the success rates for individual objects. The results show that the accelerometer data of the IMUs has the largest contribution to the overall grasp prediction, which we attribute to its ability to detect precise movements of the gripper during grasping.

ICRA Conference 2018 Conference Paper

Affordance-Based Multi-Contact Whole-Body Pose Sequence Planning for Humanoid Robots in Unknown Environments

Peter Kaiser 0001
Christian Mandery
Andreas Boltres
Tamim Asfour

Despite impressive advances of humanoid robotics, the autonomous planning of whole-body loco-manipulation actions in unknown environments is still an open problem. In our previous work, we addressed two fundamental aspects related to this problem: 1) the autonomous detection of end-effector contact opportunities in unknown environments and 2) the goal-directed planning of multi-contact pose sequences, which can serve as the starting point for motion planning and control approaches of reduced complexity. Both problems suffer from the extensive amounts of possible solutions, particularly due to the complexity of humanoid robots and the multitude of available contact opportunities. In this paper, we propose a method for the planning of whole-body multi-contact tasks based on our previous work on vision-based detection of loco-manipulation affordances and whole-body multi-contact pose sequence planning. We demonstrate a combined approach for planning multi-contact pose sequences with a focus on the utilization of available end-effectors for stabilizing contacts with the environment during loco-manipulation tasks. The method is evaluated in simulation in multiple exemplary scenarios based on actual sensor data and the humanoid robot ARMAR-4.

IROS Conference 2018 Conference Paper

Coupling Mobile Base and End-Effector Motion in Task Space

Tim Welschehold
Christian Dornhege
Fabian Paus
Tamim Asfour
Wolfram Burgard

Dynamic systems are a practical alternative to motion planning in executing robot actions. They are of particular interest in Learning from Demonstration, as here we aim to carry out actions in a certain fashion, without a model or in-depth knowledge about the world, which might be difficult to achieve with a planner. Using model-based dynamic systems in task space enables robots to flexibly reproduce demonstrated actions. Nevertheless, when dealing with mobile manipulators, we face the challenge of including the kinematic constraints of the robot in the action models. In this paper we propose to couple robot base and end-effector motions generated by arbitrary dynamical systems modulating the base velocity, while respecting the robots kinematic design. To this end we learn an approximation of the inverse reachability in closed form. In real-world robot experiments we demonstrate that we are able to maintain kinematically feasible trajectories in the presence of obstacles and in configurations differing profoundly from the training scene.

IROS Conference 2018 Conference Paper

Exploration and Reconstruction of Unknown Objects using a Novel Normal and Contact Sensor

Simon Ottenhaus
Pascal Weiner
Lukas Kaul
Andreea Tulbure
Tamim Asfour

Tactile sensing of surface normals is essential for exploration of unknown objects. Many tactile sensors have been developed for contact measurement. However, few of these sensors provide surface orientation, and only up to a limited degree. This paper presents a novel contact and surface orientation sensor concept and its application for surface reconstruction of unknown objects. The sensor is comprised of an Inertial Measurement Unit (IMU) and a pressure sensor to accurately estimate the surface orientation in a wide range, while at the same time measuring contact force. We describe the developed sensor prototype and evaluate its performance regarding contact detection capability and normal estimation accuracy. We use this to reconstruct the surface of unknown objects using the humanoid robot ARMAR-III resulting in a mean reconstruction accuracy of 3. 6 mm.

ICRA Conference 2018 Conference Paper

Grasping of Unknown Objects Using Deep Convolutional Neural Networks Based on Depth Images

Philipp Schmidt 0004
Nikolaus Vahrenkamp
Mirko Wächter
Tamim Asfour

We present a data-driven, bottom-up, deep learning approach to robotic grasping of unknown objects using Deep Convolutional Neural Networks (DCNNs). The approach uses depth images of the scene as its sole input for synthesis of a single-grasp solution during execution, adequately portraying the robot's visual perception during exploration of a scene. The training input consists of precomputed high-quality grasps, generated by analytical grasp planners, accompanied with rendered depth images of the training objects. In contrast to previous work on applying deep learning techniques to robotic grasping, our approach is able to handle full end-effector poses and therefore approach directions other than the view direction of the camera. Furthermore, the approach is not limited to a certain grasping setup (e. g. parallel jaw gripper) by design. We evaluate the method regarding its force-closure performance in simulation using the KIT and YCB object model datasets as well as a big data grasping database. We demonstrate the performance of our approach in qualitative grasping experiments on the humanoid robot ARMAR-III.

IROS Conference 2018 Conference Paper

Human Motion Classification Based on Multi-Modal Sensor Data for Lower Limb Exoskeletons

Jonas Beil
Isabel Ehrenberger
Clara Scherer
Christian Mandery
Tamim Asfour

Intuitive exoskeleton control is fundamental since it contributes to improved user acceptance and wearability comfort. This requires the detection of user's motion intention and its incorporation into the exoskeleton control system. In this work, we propose a classification system based on Hidden Markov Models (HMMs), which facilitates the online classification of multi-modal sensor data acquired from a lower-limb exoskeleton based on previously defined motion patterns. For classification of these motion patterns at each time step, we consider the most recent sensor measurements by using a sliding window approach. We collected a training data set from a total number of 10 subjects performing 13 different motions with a passive exoskeleton equipped with 7 3D-force sensors and 3 inertial measurement units (IMUs). Our evaluation includes an analysis of the time needed for correct classification (latency), a validation for a training set containing all subjects and a leave-one-out validation to assess the generalization performance of the approach. The results indicate that our approach can classify motions of subjects included in the training set with an average accuracy of 92. 80% and is able to achieve a generalization performance of 84. 46%. With the selected parameters an average latency of 368. 97 ms is achieved.

IROS Conference 2018 Conference Paper

The KIT Prosthetic Hand: Design and Control

Pascal Weiner
Julia Starke
Felix Hundhausen
Jonas Beil
Tamim Asfour

The development and control of prosthetic hands is an active research area and recently progress in mechatronics, sensor integration and innovative control has been made. However, integration of different components into a prosthetic hand remains challenging due to space constraints, the requirements regarding holistic integration and the need for a user interface. In this paper, we present the KIT prosthetic hand, a novel five-finger 3D printed hand prosthesis, with its underactuated mechanism, sensors and embedded control system. The hand mechanics is based on the underactuated TUAT/Karlsruhe mechanism with two motors actuating 10 degrees of freedom. The mechanism has been realized in 3D printing technologies to facilitate a personalization of the prosthetic hand in terms of size and kinematic parameters. The prosthesis has been designed as a 50 th percentile male hand. It integrates an advanced embedded system as well as an RGB camera in the base of the palm and a colour display in the back of the hand. Experiments indicate a finger tip force of 7. 48 N to 11. 82 N, a hook grasp force of 120 N and a hand closing time of ~ 1. 3 s.

IROS Conference 2018 Conference Paper

The KIT Swiss Knife Gripper for Disassembly Tasks: A Multi-Functional Gripper for Bimanual Manipulation with a Single Arm

Júlia Borràs 0001
Raphael Heudorfer
Samuel Rader
Peter Kaiser 0001
Tamim Asfour

This work presents the concept of a robotic gripper designed for the disassembly of electromechanical devices that comprises several innovative ideas. Novel concepts include the ability to interchange built-in tools without the need to grasp them, the ability to reposition grasped objects in-hand, the capability of performing classic dual arm manipulation within the gripper and the utilization of classic industrial robotic arms kinematics within a robotic gripper. We analyze state of the art grippers and robotic hands designed for dexterous in-hand manipulation and extract common characteristics and weak points. The presented concept is obtained from the task requirements for disassembly of electromechanical devices and it is then evaluated for general purpose grasping, in-hand manipulation and operations with tools. We further present the CAD design for a first prototype.

IROS Conference 2017 Conference Paper

A combined approach for robot placement and coverage path planning for mobile manipulation

Fabian Paus
Peter Kaiser 0001
Nikolaus Vahrenkamp
Tamim Asfour

Robotic coverage path planning describes the problem of determining a configuration space trajectory for successively covering a specified workspace target area with the robot's end-effector. Performing coverage path planning for mobile robots further requires solving the problem of robot placement, i. e. determining of suitable robot base positions to perform the task. Finding an optimal solution is hard as both problems cannot be solved independently. Combined robot placement and coverage planning is particularly interesting if repositioning of the robot is costly or if simultaneous repositioning and end-effector motion is not desired. In this paper, we present a general approach for combined robot placement and coverage path planning that takes constraints like collision avoidance and static stability into account. In contrast to related approaches, we focus on mobile manipulation tasks that require a fixed placement for executing coverage trajectory segments. The approach is evaluated in two scenarios that exemplify the broad range of possible applications: The coverage of a building facade using a robotic manlift and the coverage of an industrial conveyer belt for maintenance tasks using the humanoid robot ARMAR-III.

IROS Conference 2017 Conference Paper

Autonomous view selection and gaze stabilization for humanoid robots

Markus Grotz
Timothee Habra
Renaud Ronsse
Tamim Asfour

To increase the autonomy of humanoid robots, the visual perception must support the efficient collection and interpretation of visual scene cues by providing task-dependent information. Active vision systems allow to extend the observable workspace by employing active gaze control, i. e. by shifting the gaze to relevant areas in the scene. When moving the eyes, stabilization of the camera images is crucial for successful task execution. In this paper, we present an active vision system for task-oriented selection of view directions and gaze stabilization to enable a humanoid robot to robustly perform vision-based tasks. We investigate the interaction between a gaze stabilization controller and view planning to select the next best view direction based on saliency maps which encode task-relevant information. We demonstrate the performance of the systems in a real world scenario, in which a humanoid robot is performing vision-based grasping while moving, a task that would not be possible without the combination of view selection and gaze stabilization.

IROS Conference 2017 Conference Paper

Task-oriented generalization of dynamic movement primitive

You Zhou 0007
Tamim Asfour

An important question in imitation learning is how to generalize a learned motion to novel situations. The motion generalization depends on a set of features which can be represented as feature vectors spanning a feature space, called query space. The purpose of generalization is to find a mapping from this query space to the motion primitive space (MP space). In this paper, we address the problem of generalization of dynamic movement primitives (DMPs) to new queries by applying locally weighted regression (LWR) with radial basis functions (RBF). Since two DMPs differ only in their non-linear part, we transform the problem of DMP generalization to a regression analysis problem. We introduce a task-oriented regression algorithm with a cost function that takes task constraints into consideration and which relies on model switching to solve the problem of poor DMP generalization when using a single regression model for the entire query space. The evaluation shows that our algorithm outperforms related approaches in the literature in terms of generalization capabilities.

IROS Conference 2016 Conference Paper

Coordinate Change Dynamic Movement Primitives - A leader-follower approach

You Zhou 0007
Martin Do
Tamim Asfour

Dynamic movement primitives prove to be a useful and effective way to represent a movement of a given agent. However, the original DMP formulation does not take the interaction among multiple agents into the consideration. Thus, many researchers focus on the development of a coupling term for the underlying dynamical system and its associated learning strategies. The result is highly dependent on the quality of the learning methods. In this paper, we present a new way to formulate and realize interactive movement primitive in a leader-follower configuration, where the relationship between the follower and the leader is explicitly represented via the new formulation. This new formulation does not only simplify the learning process, but it also meets the requirements of several applications. We separately tested our new formulation in the context of the handover task and the wiping task. The results prove the flexibility and simplicity of the new formulation.

IROS Conference 2016 Conference Paper

Heuristic 3D object shape completion based on symmetry and scene context

David Schiebener
Andreas Schmidt 0002
Nikolaus Vahrenkamp
Tamim Asfour

Object shape information is essential for robot manipulation tasks, in particular for grasp planning and collision-free motion planning. But in general a complete object model is not available, in particular when dealing with unknown objects. We propose a method for completing shapes that are only partially known, which is a common situation when a robot perceives a new object only from one direction. Our approach is based on the assumption that most objects used in service robotic setups have symmetries. We determine and rate symmetry plane candidates to estimate the hidden parts of the object. By finding possible supporting planes based on its immediate neighborhood, the search space for symmetry planes is restricted, and the bottom part of the object is added. Gaps along the sides in the direction of the view axis are closed by linear interpolation. We evaluate our approach with real-world experiments using the YCB object and model set [1].

ICRA Conference 2016 Conference Paper

Resource-aware motion planning

Manfred Kröhnert
Raphael Grimm
Nikolaus Vahrenkamp
Tamim Asfour

We address the question of how resource-aware concepts can be utilized in motion planning algorithms. Resource-awareness facilitate better resource allocation on global system level, e. g. when a humanoid robot needs to distribute and schedule a wide variety of concurrent algorithms. We present a motion planning approach that employs self-monitoring concepts in order to identify the difficulty of the planning problem. Resources are requested dynamically and adapted based on problem difficulty and current planning progress. We show how dynamic adaptation of resource allocation on algorithmic level can reduce the system workload as compared to static resource allocation while meeting Quality of Service (QoS) measures such as average workload or efficiency. We evaluate our approach both in several synthetic setups with varying difficulty and with the humanoid robot ARMAR-4.

IROS Conference 2016 Conference Paper

Towards a hierarchy of loco-manipulation affordances

Peter Kaiser 0001
Eren Erdal Aksoy
Markus Grotz
Tamim Asfour

We propose a formalism for the hierarchical representation of affordances. Starting with a perceived model of the environment consisting of geometric primitives like planes or cylinders, we define a hierarchical system for affordance extraction whose foundation are elementary power grasp affordances. Higher-level affordances, e. g. bimanual affordances, result from combining lower-level affordances with additional properties concerning the underlying geometric primitives of the scene. We model affordances as continuous certainty functions taking into account properties of the environmental elements and the perceiving robot's embodiment. The developed formalism is regarded as the basis for the description of whole-body affordances, i. e. affordances associated with whole-body actions. The proposed formalism was implemented and experimentally evaluated in multiple scenarios based on RGB-D camera data. The feasibility of the approach is demonstrated on a real robotic platform.

IROS Conference 2016 Conference Paper

Using language models to generate whole-body multi-contact motions

Christian Mandery
Júlia Borràs 0001
Mirjam Jöchner
Tamim Asfour

We present a novel approach for generating sequences of whole-body poses with multi-contacts for humanoid robots, which is inspired by techniques from natural language processing. To this end, we propose a probabilistic n-gram language model learned from observation of human locomotion tasks. Human motion data is automatically segmented according to detected contacts of the body with the environment to provide support, that is, support poses, which are further subdivided with regard to whole-body configuration. These poses are subsequently used to train a language model, whose words are the poses, and whose sentences represent sequences of poses. Then, we propose a planning algorithm that, given the constraints imposed by a task, finds the sequence of transitions with the highest probability according to our language model. We have applied our approach to 140 motion capture recordings of locomotion tasks that involve using one or both hands for support. The evaluation demonstrates that our approach is able to generate complex sets of pose transitions, and shows promising results regarding its application to more complex tasks.

ICRA Conference 2015 Conference Paper

A jumping robot using soft pneumatic actuator

Feng Ni
Daniel Rojas
Kai Tang 0001
Lilong Cai
Tamim Asfour

This paper presents the development of a new type of robot capable of vertical and directional jumping. The robot uses soft silicone elastomer based pneumatic actuators as legs that accelerate the platform upwards by rapid pressurization. The robot is able to control and adjust the direction of the jumping by altering the timing patterns in which the individual legs are activated.

IROS Conference 2015 Conference Paper

A whole-body pose taxonomy for loco-manipulation tasks

Júlia Borràs 0001
Tamim Asfour

Exploiting interaction with the environment is a promising and powerful way to enhance stability of humanoid robots and robustness while executing locomotion and manipulation tasks. Recently some works have started to show advances in this direction considering humanoid locomotion with multi-contacts, but to be able to fully develop such abilities in a more autonomous way, we need to first understand and classify the variety of possible poses a humanoid robot can achieve to balance. To this end, we propose the adaptation of a successful idea widely used in the field of robot grasping to the field of humanoid balance with multi-contacts: a whole-body pose taxonomy classifying the set of whole-body robot configurations that use the environment to enhance stability. We have revised criteria of classification used to develop grasping taxonomies, focusing on structuring and simplifying the large number of possible poses the human body can adopt. We propose a taxonomy with 46 poses, containing three main categories, considering number and type of supports as well as possible transitions between poses. The taxonomy induces a classification of motion primitives based on the pose used for support, and a set of rules to store and generate new motions. We present preliminary results that apply known segmentation techniques to motion data from the KIT whole-body motion database. Using motion capture data with multi-contacts, we can identify support poses providing a segmentation that can distinguish between locomotion and manipulation parts of an action.

ICRA Conference 2015 Conference Paper

Kinodynamic randomized rearrangement planning via dynamic transitions between statically stable states

Joshua A. Haustein
Jennifer E. King
Siddhartha S. Srinivasa
Tamim Asfour

In this work we present a fast kinodynamic RRT-planner that uses dynamic nonprehensile actions to rearrange cluttered environments. In contrast to many previous works, the presented planner is not restricted to quasi-static interactions and monotonicity. Instead the results of dynamic robot actions are predicted using a black box physics model. Given a general set of primitive actions and a physics model, the planner randomly explores the configuration space of the environment to find a sequence of actions that transform the environment into some goal configuration.

ICRA Conference 2015 Conference Paper

Nonprehensile whole arm rearrangement planning on physics manifolds

Jennifer E. King
Joshua A. Haustein
Siddhartha S. Srinivasa
Tamim Asfour

We present a randomized kinodynamic planner that solves rearrangement planning problems. We embed a physics model into the planner to allow reasoning about interaction with objects in the environment. By carefully selecting this model, we are able to reduce our state and action space, gaining tractability in the search. The result is a planner capable of generating trajectories for full arm manipulation and simultaneous object interaction. We demonstrate the ability to solve more rearrangement by pushing tasks than existing primitive based solutions. Finally, we show the plans we generate are feasible for execution on a real robot.

IROS Conference 2014 Conference Paper

Changing pre-grasp strategies with increasing object location uncertainty

Boris Illing
Tamim Asfour
Nancy S. Pollard

Successful and robust grasping for humanoid robots is still an ongoing research topic in robotics. Applying human-inspired grasping strategies does not only correspond with more natural looking motions but can also yield good results regarding task success when having to deal with uncertainty. This study investigates human high-level grasping strategies and how they tend to change for different objects when the uncertainty of object location or orientation increases in between two grasps. We are especially interested in potential gains for humanoid robots in a common household setting. By analyzing collected data from human subject grasp experiments with a set of typical objects found in people's homes, we get better insight into how humans handle uncertainty, as well as when and how they change their applied pre-grasp strategy. By adapting the by far most often observed change from a direct grasp attempt to a tapping strategy when dealing with high uncertainty, we can demonstrate a substantial increase of grasp success rate for our robot system with a Shadow Dexterous Hand mounted on a Motoman SDA10 robot while using less than two hand correction steps on average.

ICRA Conference 2014 Conference Paper

Extracting common sense knowledge from text for robot planning

Peter Kaiser 0001
Mike Lewis
Ronald P. A. Petrick
Tamim Asfour
Mark Steedman

Autonomous robots often require domain knowledge to act intelligently in their environment. This is particularly true for robots that use automated planning techniques, which require symbolic representations of the operating environment and the robot's capabilities. However, the task of specifying domain knowledge by hand is tedious and prone to error. As a result, we aim to automate the process of acquiring general common sense knowledge of objects, relations, and actions, by extracting such information from large amounts of natural language text, written by humans for human readers. We present two methods for knowledge acquisition, requiring only limited human input, which focus on the inference of spatial relations from text. Although our approach is applicable to a range of domains and information, we only consider one type of knowledge here, namely object locations in a kitchen environment. As a proof of concept, we test our approach using an automated planner and show how the addition of common sense knowledge can improve the quality of the generated plans.

ICRA Conference 2014 Conference Paper

Learn to wipe: A case study of structural bootstrapping from sensorimotor experience

Martin Do
Julian Schill
Johannes Ernesti
Tamim Asfour

In this paper, we address the question of generative knowledge construction from sensorimotor experience, which is acquired by exploration. We show how actions and their effects on objects, together with perceptual representations of the objects, are used to build generative models which then can be used in internal simulation to predict the outcome of actions. Specifically, the paper presents an experiential cycle for learning association between object properties (softness and height) and action parameters for the wiping task and building generative models from sensorimotor experience resulting from wiping experiments. Object and action are linked to the observed effect to generate training data for learning a non-parametric continuous model using Support Vector Regression. In subsequent iterations, this model is grounded and used to make predictions on the expected effects for novel objects which can be used to constrain the parameter exploration. The cycle and skills have been implemented on the humanoid platform ARMAR-IIIb. Experiments with set of wiping objects differing in softness and height demonstrate efficient learning and adaptation behavior of action of wiping.

ICRA Conference 2014 Conference Paper

Physical interaction for segmentation of unknown textured and non-textured rigid objects

David Schiebener
Ales Ude
Tamim Asfour

We present an approach for autonomous interactive object segmentation by a humanoid robot. The visual segmentation of unknown objects in a complex scene is an important prerequisite for e. g. object learning or grasping, but extremely difficult to achieve through passive observation only. Our approach uses the manipulative capabilities of humanoid robots to induce motion on the object and thus integrates the robots manipulation and sensing capabilities to segment previously unknown objects. We show that this is possible without any human guidance or pre-programmed knowledge, and that the resulting motion allows for reliable and complete segmentation of new objects in an unknown and cluttered environment. We extend our previous work, which was restricted to textured objects, by devising new methods for the generation of object hypotheses and the estimation of their motion after being pushed by the robot. These methods are mainly based on the analysis of motion of color annotated 3D points obtained from stereo vision, and allow the segmentation of textured as well as non-textured rigid objects. In order to evaluate the quality of the obtained segmentations, they are used to train a simple object recognizer. The approach has been implemented and tested on the humanoid robot ARMAR-III, and the experimental results confirm its applicability on a wide variety of objects even in highly cluttered scenes.

ICRA Conference 2014 Conference Paper

Robust real-time 6D active visual localization for humanoid robots

David Israel Gonzalez-Aguirre
Michael Vollert
Tamim Asfour
Rüdiger Dillmann

Overcoming the perceptual limitations of hu-manoid robots requires representations exploitable by highly integrable simulation, sensing, planning and acting components. Therefore, a novel active visual localization component for humanoid robots based on particle filtering in CAD environments is introduced. Specifically, two new components are presented: i) A vector-graphics prediction method employing hierarchical CAD environmental representations is presented. ii) A gaze attention method within the prediction-update cycle of the particle filter increases the available amount of visual features for localization while allowing adjustable task coupling. Finally, large and unobstructive ground-truth validation with the humanoid robot ARMAR-IIIb [1] in a made-for-humans environment shows the robustness, accuracy and performance of the proposed methods.

IROS Conference 2013 Conference Paper

Development of a five-finger dexterous hand without feedback control: The TUAT/Karlsruhe humanoid hand

Naoki Fukaya
Tamim Asfour
Rüdiger Dillmann
Shigeki Toyama

In order to realize performance gain of a robot or an artificial arm, the end-effector which exhibits the same function as human beings and can respond to various objects and environment needs to be realized. Then, we developed the new hand which paid its attention to the structure of human being's hand which realize operation in human-like manipulation (called TUAT/Karlsruhe Humanoid Hand). Since this humanoid hand has the structure of adjusting grasp shape and grasp force automatically, it does not need a touch sensor and feedback control. It is designed for the humanoid robot which has to work autonomously or interactively in cooperation with humans and for an artificial arm for handicapped persons. The ideal end-effectors for such an artificial arm or a humanoid would be able to use the tools and objects that a person uses when working in the same environment. If this humanoid hand can operate the same tools, a machine and furniture, it may be possible to work under the same environment as human beings. As a result of adopting a new function of a palm and the thumb, the robot hand could do the operation which was impossible until now. The humanoid hand realized operations which hold a kitchen knife, grasping a fan, a stick, uses the scissors and uses chopsticks.

ICRA Conference 2013 Conference Paper

Gaze selection during manipulation tasks

Kai Welke
David Schiebener
Tamim Asfour
Rüdiger Dillmann

A major strength of humanoid robotics platforms consists in their potential to perform a wide range of manipulation tasks in human-centered environments thanks to their anthropomorphic design. Further, they offer active head-eye systems which allow to extend the observable workspace by employing active gaze control. In this work, we address the question where to look during manipulation tasks while exploiting these two key capabilities of humanoid robots. We present a solution to the gaze selection problem, which takes into account constraints derived from manipulation tasks. Thereby, three different subproblems are addressed: the representation of the acquired visual input, the calculation of saliency based on this representation, and the selection of the most suitable gaze direction. As representation of the visual input, a probabilistic environmental model is discussed, which allows to take into account the dynamic nature of manipulation tasks. At the core of the gaze selection mechanism, a novel saliency measure is proposed that includes accuracy requirements from the manipulation task in the saliency calculation. Finally, an iterative procedure based on spherical graphs is developed in order to decide for the best gaze direction. The feasibility of the approach is experimentally evaluated in the context of bimanual manipulation tasks on the humanoid robot ARMAR-III.

IROS Conference 2013 Conference Paper

Modulation of motor primitives using force feedback: Interaction with the environment and bimanual tasks

Andrej Gams
Bojan Nemec
Leon Zlajpah
Mirko Wächter
Auke Jan Ijspeert
Tamim Asfour
Ales Ude

The framework of dynamic movement primitives allows the generation of discrete and periodic trajectories, which can be modulated in various aspects. We propose and evaluate a novel modulation approach that includes force feedback and thus allows physical interaction with objects and the environment. The proposed approach also enables the coupling of independently executed robotic trajectories, simplifying the execution of bimanual and tightly coupled cooperative tasks. We apply an iterative learning control algorithm to learn a coupling term, which is applied to the original trajectory in a feed-forward fashion. The coupling term modifies the trajectory in accordance to either the desired position or external force. The strengths of the approach are shown in bimanual or two-agent obstacle avoidance tasks, where no higher level cognitive reasoning or planning are required. Results of simulated and real-world experiments on the ARMAR-III humanoid robot in interaction and object lifting tasks, and on two KUKA LWR robots in a bimanual setting are presented.

IROS Conference 2013 Conference Paper

Optimal high-dynamic-range image acquisition for humanoid robots

David Israel Gonzalez-Aguirre
Tamim Asfour
Rüdiger Dillmann

Humanoid robots should be able to visually recognize objects and estimate their 6D pose in real environmental conditions with their limited sensor capabilities. In order to achieve these visual skills, it is necessary to establish an optimal visual transducer connecting the scene layout with the internal representations of objects and places. This visual transducer should capture the noiseless visual manifold of the scene with high-dynamic-range in an efficient manner. Our endeavor is to develop such a visual transducer using the widespread LDR cameras in humanoid robots. In our previous work, the noiseless acquisition of continuous images [1] and the improved radio-metric calibration [2] already enabled the humanoid robots to attain the desired visual manifold in terms of quality. However, since the radiance range of the scene can be very wide, the required amount of exposures to capture the visual manifold (robustly without radiance inconsistencies) turns impractically large in terms of scope, granularity and acquisition time. In this article, a method for estimating the minimal amount of exposures and their particular integration times is presented. This method integrates our previous work in order to synthesize HDR images with the minimal amount of exposures while ensuring the high quality of the resulting image. Conclusively, the minimal exposure set provides performance improvements without quality trade-off. Experimental evaluation is presented with the humanoid robots ARMAR-III a, b [3].

ICRA Conference 2013 Conference Paper

Robot placement based on reachability inversion

Nikolaus Vahrenkamp
Tamim Asfour
Rüdiger Dillmann

Having a representation of the capabilities of a robot is helpful when online queries, such as solving the inverse kinematics (IK) problem for grasping tasks, must be processed efficiently in the real world. When workspace representations, e. g. the reachability of an arm, are considered, additional quality information such as manipulability or self-distance can be employed to enrich the spatial data. In this work we present an approach of inverting such precomputed reachability representations in order to generate suitable robot base positions for grasping. Compared to existing works, our approach is able to generate a distribution in SE(2), the cross-space consisting of 2D position and 1D orientation, that describes potential robot base poses together with a quality index. We show how this distribution can be queried quickly in order to find oriented base poses from which a target grasping pose is reachable without collisions. The approach is evaluated in simulation using the humanoid robot ARMAR-III [1] and an extension is presented that allows to find suitable base poses for trajectory execution.

ICRA Conference 2013 Conference Paper

Synthesizing object receiving motions of humanoid robots with human motion database

Katsu Yamane
Marcel Revfi
Tamim Asfour

This paper presents a method for synthesizing motions of a humanoid robot that receives an object from a human, with focus on a natural object passing scenario where the human initiates the passing motion by moving an object towards the robot, which continuously adapts its motion to the observed human motion in real time. In this scenario, the robot not only has to recognize and adapt to the human action but also has to synthesize its motion quickly so that the human does not have to wait holding an object. We solve these issues by using a human motion database obtained from two persons performing the object passing task. The rationale behind this approach is that human performance of such a simple task is repeatable, and therefore the receiver (robot) motion can be synthesized by looking up the passer motion in a database. We demonstrate in simulation that the robot can start extending the arm at an appropriate timing and take hand configurations suitable for the object being passed. We also perform hardware experiments of object handing from a human to a robot.

IROS Conference 2013 Conference Paper

Towards online trajectory generation considering robot dynamics and torque limits

Robert K. Katzschmann
Torsten Kröger
Tamim Asfour
Oussama Khatib

Generating robot motion trajectories instantaneously in the moment unforeseen sensor events happen is very essential for many real-world robot applications. Using a previous work on online trajectory generation as a basis, this paper proposes an alternative approach that also considers dynamic models. The former class of algorithms does not take into account dynamically changing acceleration capabilities based on maximum actuator forces/torques. This paper extends target velocity-based algorithms of the previous approach by taking into consideration the entire system dynamics when generating trajectories online within one control cycle (typically 1 ms or less). The extension includes the acceleration capabilities of a robot at every discrete time step assuming constant values for the maximum actuator forces/torques, thus allowing the generation of adaptive trajectory profiles during the motion of the robot. Several real-world experimental results using a seven-degree-of-freedom lightweight robot arm underline the relevance of this extension.

ICRA Conference 2012 Conference Paper

Constellation - An algorithm for finding robot configurations that satisfy multiple constraints

Peter Kaiser 0001
Dmitry Berenson
Nikolaus Vahrenkamp
Tamim Asfour
Rüdiger Dillmann
Siddhartha S. Srinivasa

Planning motion for humanoid robots requires obeying simultaneous constraints on balance, collision-avoidance, and end-effector pose, among others. Several algorithms are able to generate configurations that satisfy these constraints given a good initial guess, i. e. a configuration which is already close to satisfying the constraints. However, when selecting goals for a planner a close initial guess is rarely available. Methods that attempt to satisfy all constraints through direct projection from a distant initial guess often fail due to opposing gradients for the various constraints, joint-limits, or singularities. We approach the problem of generating a constrained goal by searching for a configuration in the intersection of all constraint manifolds in configuration space (C-space). Starting with an initial guess, our algorithm, Constellation, builds a graph in C-space whose nodes are configurations that satisfy one or more constraints and whose cycles determine where the algorithm explores next. We compare the performance of our approach to direct projection and a previously-proposed cyclic projection method on reaching tasks for a humanoid robot with 33 DOF. We find that Constellation performs the best in terms of the number of solved queries across a wide range of problem difficulty. However, this success comes at higher computational cost.

IROS Conference 2012 Conference Paper

Learning robot dynamics with Kinematic Bézier Maps

Stefan Ulbrich
Michael Garrett Bechtel
Tamim Asfour
Rüdiger Dillmann

The previously presented Kinematic Bézier Maps (KBM) are a machine learning algorithm that has been tailored to efficiently learn the kinematics of redundant robots. This algorithm relies upon a representation based on projective geometry that uses a special set of polynomial functions borrowed from the field of Computer Aided Geometric Design (CAGD). So far, it has only been possible to learn a model of the forward kinematics function. In this paper, we show how the KBM algorithm can be modified to learn the robot's equation of motion and, hence, its inverse dynamic model. Results from experiments with a simulated serial robot manipulator are presented that clearly show the advantages of our approach compared to general function approximation methods.

ICRA Conference 2012 Conference Paper

Template-based learning of grasp selection

Alexander Herzog
Peter Pastor
Mrinal Kalakrishnan
Ludovic Righetti
Tamim Asfour
Stefan Schaal

The ability to grasp unknown objects is an important skill for personal robots, which has been addressed by many present and past research projects, but still remains an open problem. A crucial aspect of grasping is choosing an appropriate grasp configuration, i. e. the 6d pose of the hand relative to the object and its finger configuration. Finding feasible grasp configurations for novel objects, however, is challenging because of the huge variety in shape and size of these objects. Moreover, possible configurations also depend on the specific kinematics of the robotic arm and hand in use. In this paper, we introduce a new grasp selection algorithm able to find object grasp poses based on previously demonstrated grasps. Assuming that objects with similar shapes can be grasped in a similar way, we associate to each demonstrated grasp a grasp template. The template is a local shape descriptor for a possible grasp pose and is constructed using 3d information from depth sensors. For each new object to grasp, the algorithm then finds the best grasp candidate in the library of templates. The grasp selection is also able to improve over time using the information of previous grasp attempts to adapt the ranking of the templates. We tested the algorithm on two different platforms, the Willow Garage PR2 and the Barrett WAM arm which have very different hands. Our results show that the algorithm is able to find good grasp configurations for a large set of objects from a relatively small set of demonstrations, and does indeed improve its performance over time.

ICRA Conference 2011 Conference Paper

6-DoF model-based tracking of arbitrarily shaped 3D objects

Pedram Azad
David Münch
Tamim Asfour
Rüdiger Dillmann

Image-based 6-DoF pose estimation of arbitrarily shaped 3D objects based on their shape is a rarely studied problem. Most existing image-based methods for pose estimation either exploit textural information in form of local features or, if shape-based, rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but in practice assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on its relative pose to the observing camera. One way to reduce the solution space is to exploit temporal information, i. e. perform tracking. Again, existing model-based tracking approaches rely on relatively simple object geometries. In this paper, we propose a particle filter based tracking approach that can deal with arbitrary shapes and arbitrary or even no texture, i. e. it offers a general solution to the rigid object tracking problem. As our approach can deal with occlusions, it is in particular of interest in the context of goal-directed imitation learning involving the observation of object manipulations. Results of simulation experiments as well as real-world experiments with different object types prove the practical applicability of our approach.

IROS Conference 2011 Conference Paper

Planning grasps for robotic hands using a novel object representation based on the medial axis transform

Markus Przybylski
Tamim Asfour
Rüdiger Dillmann

We introduce an approach for enabling sampling-based planners to compute motions with humanlike appearance. The proposed method is based on a space of blendable example motions collected by motion capture. This space is explored by a sampling-based planner that is able to produce motions around obstacles while keeping solutions similar to the original examples. The results therefore largely maintain the humanlike characteristics observed in the example motions. The method is applied to generic upper-body actions and is complemented by a locomotion planner that searches for suitable body placements for executing upper-body actions successfully. As a result, our overall multi-modal planning method is able to automatically coordinate whole-body motions for action execution among obstacles, and the produced motions remain similar to example motions given as input to the system.

ICRA Conference 2011 Conference Paper

RDT +: A parameter-free algorithm for exact motion planning

Nikolaus Vahrenkamp
Peter Kaiser 0001
Tamim Asfour
Rüdiger Dillmann

In this paper parameter-free concepts for exact motion planning are investigated. With the proposed RDT + approach the collision detection parameters of a Rapidly exploring Dense Tree (RDT) are automatically adjusted until an exact solution can be found. For efficient planning discrete collision detection routines are used within the RDT planner and by verifying the results with exact collision detection methods, the RDT+ concept allows to compute motions that are guaranteed collision-free. We show the probabilistic completeness of the proposed planner and present an extension for handling narrow passages. The algorithms are evaluated in different experiments, including narrow passages and high-dimensional planning problems, that are solved in simulation and on the humanoid robot ARMAR-III.

IROS Conference 2011 Conference Paper

The OpenGRASP benchmarking suite: An environment for the comparative analysis of grasping and dexterous manipulation

Stefan Ulbrich
Daniel Kappler
Tamim Asfour
Nikolaus Vahrenkamp
Alexander Bierbaum
Markus Przybylski
Rüdiger Dillmann

In this work, we present a new software environment for the comparative evaluation of algorithms for grasping and dexterous manipulation. The key aspect in its development is to provide a tool that allows the reproduction of well-defined experiments in real-life scenarios in every laboratory and, hence, benchmarks that pave the way for objective comparison and competition in the field of grasping. In order to achieve this, experiments are performed on a sound open-source software platform with an extendable structure in order to be able to include a wider range of benchmarks defined by robotics researchers. The environment is integrated into the OpenGRASP toolkit that is built upon the OpenRAVE project and includes grasp-specific extensions and a tool for the creation/integration of new robot models. Currently, benchmarks for grasp and motion planningare included as case studies, as well as a library of domestic everyday objects models, and a real-life scenario that features a humanoid robot acting in a kitchen.

ICRA Conference 2011 Conference Paper

Towards a unifying grasp representation for imitation learning on humanoid robots

Martin Do
Tamim Asfour
Rüdiger Dillmann

In this paper, we present a grasp representation in task space exploiting position information of the fingertips. We propose a new way for grasp representation in the task space, which provides a suitable basis for grasp imitation learning. Inspired by neuroscientific findings, finger movement synergies in the task space together with fingertip positions are used to derive a parametric low-dimensional grasp representation. Taking into account correlating finger movements, we describe grasps using a system of virtual springs to connect the fingers, where different grasp types are defined by parameterizing the spring constants. Based on such continuous parameterization, all instantiation of grasp types and all hand preshapes during a grasping action (reach, preshape, enclose, open) can be represented. We present experimental results, in which the spring constants are merely estimated from fingertip motion tracking using a stereo camera setup of a humanoid robot. The results show that the generated grasps based on the proposed representation are similar to the observed grasps.

ICRA Conference 2011 Conference Paper

Towards shape-based visual object categorization for humanoid robots

David Israel Gonzalez-Aguirre
Julian Hoch
Sebastian Röhl
Tamim Asfour
Eduardo Bayro-Corrochano
Rüdiger Dillmann

Humanoid robots should be able to grasp and handle objects in the environment, even if the objects are seen for the first time. A plausible solution to this problem is to categorize these objects into existing classes with associated actions and functional knowledge. So far, efforts on visual object categorization using humanoid robots have either been focused on appearance-based methods or have been restricted to object recognition without generalization capabilities. In this work, a shape model-based approach using stereo vision and machine learning for object categorization is introduced. The state-of-the-art features for shape matching and shape retrieval were evaluated and selectively transfered into the visual categorization. Visual sensing from different vantage points allows the reconstruction of 3D mesh models of the objects found in the scene by exploiting knowledge about the environment for model-based segmentation and registration. These reconstructed 3D mesh models were used for shape feature extraction for categorization and provide sufficient information for grasping and manipulation. Finally, the visual categorization was successfully performed with a variety of features and classifiers allowing proper categorization of unknown objects even when object appearance and shape substantially differ from the training set. Experimental evaluation with the humanoid robot ARMAR-IIIa is presented.

ICRA Conference 2010 Conference Paper

Autonomous acquisition of visual multi-view object representations for object recognition on a humanoid robot

Kai Welke
Jan Issac
David Schiebener
Tamim Asfour
Rüdiger Dillmann

The autonomous acquisition of object representations which allow recognition, localization and grasping of objects in the environment is a challenging task, which has shown to be difficult. In this paper, we present a systems for autonomous acquisition of visual object representations, which endows a humanoid robot with the ability to enrich its internal object representation and allows the realization of complex visual tasks. More precisely, we present techniques for segmentation and modeling of objects held in the five-fingered robot hand. Multiple object views are generated by rotating the held objects in the robot's field of view. The acquired object representations are evaluated in the context of visual search and object recognition tasks in cluttered environments. Experimental results show successful implementation of the complete cycle from object exploration to object recognition on a humanoid robot.

ICRA Conference 2010 Conference Paper

Integrated Grasp and motion planning

Nikolaus Vahrenkamp
Martin Do
Tamim Asfour
Rüdiger Dillmann

In this work, we present an integrated planner for collision-free single and dual arm grasping motions. The proposed Grasp-RRT planner combines the three main tasks needed for grasping an object: finding a feasible grasp, solving the inverse kinematics and searching a collision-free trajectory that brings the hand to the grasping pose. Therefore, RRT-based algorithms are used to build a tree of reachable and collision-free configurations. During RRT-generation, potential grasping positions are generated and approach movements toward them are computed. The quality of reachable grasping poses is scored with an online grasp quality measurement module which is based on the computation of applied forces in order to diminish the net torque. We also present an extension to a dual arm planner which generates bimanual grasps together with corresponding dual arm grasping motions. The algorithms are evaluated with different setups in simulation and on the humanoid robot ARMAR-III.

IROS Conference 2010 Conference Paper

Unions of balls for shape approximation in robot grasping

Markus Przybylski
Tamim Asfour
Rüdiger Dillmann

Typical tasks of future service robots involve grasping and manipulating a large variety of objects differing in size and shape. Generating stable grasps on 3D objects is considered to be a hard problem, since many parameters such as hand kinematics, object geometry, material properties and forces have to be taken into account. This results in a high-dimensional space of possible grasps that cannot be searched exhaustively. We believe that the key to find stable grasps in an efficient manner is to use a special representation of the object geometry that can be easily analyzed. In this paper, we present a novel grasp planning method that evaluates local symmetry properties of objects to generate only candidate grasps that are likely to be of good quality. We achieve this by computing the medial axis which represents a 3D object as a union of balls. We analyze the symmetry information contained in the medial axis and use a set of heuristics to generate geometrically and kinematically reasonable candidate grasps. These candidate grasps are tested for force-closure. We present the algorithm and show experimental results on various object models using an anthropomorphic hand of a humanoid robot in simulation.

IROS Conference 2009 Conference Paper

Accurate shape-based 6-DoF pose estimation of single-colored objects

Pedram Azad
Tamim Asfour
Rüdiger Dillmann

The problem of accurate 6-DoF pose estimation of 3D objects based on their shape has so far been solved only for specific object geometries. Edge-based recognition and tracking methods rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on the view angle. One way to tackle this problem is to use canonical views. However, accurate shape-based 6-DoF pose estimation requires more information than matching of canonical views can provide. In this paper, we present a novel approach to 6-DoF pose estimation of single-colored objects based on their shape. Our approach combines stereo triangulation with matching against a high-resolution view set of the object, each view having associated orientation information. The errors that arise from separating the position and orientation computation in first place are corrected by a subsequent correction procedure based on online 3D model projection. The proposed approach can estimate the pose of a single object within 20 ms using conventional hardware.

ICRA Conference 2009 Conference Paper

Active multi-view object search on a humanoid head

Kai Welke
Tamim Asfour
Rüdiger Dillmann

Visual search is a common daily human activity and a prerequisite to the interaction with objects encountered in cluttered environments. Humanoid robots that are supposed to take part in human daily life should possess similar capabilities in terms of representing, attending to and recalling objects of interest in order to ensure robust perception in human-centered environments. In this paper, we present necessary processes, memories and representations which allow to identify and store locations of objects, encountered from different angles of view, in a visual search task. In particular, we introduce the so-called Feature Ego-Sphere (FES) as the scene memory for a humanoid robot. Experiments comprising different visual search tasks have been carried out on an active humanoid head equipped with perspective and foveal stereo camera systems. The scene is analyzed actively using both camera systems in order to find instances of searched objects in a consistent and persistent manner.

IROS Conference 2009 Conference Paper

Combining Harris interest points and the SIFT descriptor for fast scale-invariant object recognition

Pedram Azad
Tamim Asfour
Rüdiger Dillmann

In the recent past, the recognition and localization of objects based on local point features has become a widely accepted and utilized method. Among the most popular features are currently the SIFT features, the more recent SURF features, and region-based features such as the MSER. For time-critical application of object recognition and localization systems operating on such features, the SIFT features are too slow (500–600 ms for images of size 640×480 on a 3GHz CPU). The faster SURF achieve a computation time of 150–240 ms, which is still too slow for active tracking of objects or visual servoing applications. In this paper, we present a combination of the Harris corner detector and the SIFT descriptor, which computes features with a high repeatability and very good matching properties within approx. 20 ms. While just computing the SIFT descriptors for computed Harris interest points would lead to an approach that is not scale-invariant, we will show how scale-invariance can be achieved without a time-consuming scale space analysis. Furthermore, we will present results of successful application of the proposed features within our system for recognition and localization of textured objects. An extensive experimental evaluation proves the practical applicability of our approach.

IROS Conference 2009 Conference Paper

Humanoid motion planning for dual-arm manipulation and re-grasping tasks

Nikolaus Vahrenkamp
Dmitry Berenson
Tamim Asfour
James J. Kuffner
Rüdiger Dillmann

In this paper, we present efficient solutions for planning motions of dual-arm manipulation and re-grasping tasks. Motion planning for such tasks on humanoid robots with a high number of degrees of freedom (DoF) requires computationally efficient approaches to determine the robot's full joint configuration at a given grasping position, i. e. solving the Inverse Kinematics (IK) problem for one or both hands of the robot. In this context, we investigate solving the inverse kinematics problem and motion planning for dual-arm manipulation and re-grasping tasks by combining a gradient-descent approach in the robot's pre-computed reachability space with random sampling of free parameters. This strategy provides feasible IK solutions at a low computation cost without resorting to iterative methods which could be trapped by joint-limits. We apply this strategy to dual-arm motion planning tasks in which the robot is holding an object with one hand in order to generate whole-body robot configurations suitable for grasping the object with both hands. In addition, we present two probabilistically complete RRT-based motion planning algorithms (J+-RRT and IK-RRT) that interleave the search for an IK solution with the search for a collision-free trajectory and the extension of these planners to solving re-grasping problems. The capabilities of combining IK methods and planners are shown both in simulation and on the humanoid robot ARMAR-III performing dual-arm tasks in a kitchen environment.

ICRA Conference 2009 Conference Paper

Learning and generalization of motor skills by learning from demonstration

Peter Pastor
Heiko Hoffmann
Tamim Asfour
Stefan Schaal

We provide a general approach for learning robotic motor skills from human demonstration. To represent an observed movement, a non-linear differential equation is learned such that it reproduces this movement. Based on this representation, we build a library of movements by labeling each recorded movement according to task and context (e. g. , grasping, placing, and releasing). Our differential equation is formulated such that generalization can be achieved simply by adapting a start and a goal parameter in the equation to the desired position values of a movement. For object manipulation, we present how our framework extends to the control of gripper orientation and finger position. The feasibility of our approach is demonstrated in simulation as well as on the Sarcos dextrous robot arm. The robot learned a pick-and-place operation and a water-serving task and could generalize these tasks to novel situations.

IROS Conference 2008 Conference Paper

Adaptive motion planning for humanoid robots

Nikolaus Vahrenkamp
Christian Scheurer
Tamim Asfour
James J. Kuffner
Rüdiger Dillmann

Motion planning for robots with many degrees of freedom (DoF) is a generally unsolved problem in the robotics context. In this work an approach for trajectory planning is presented, which takes account of the different kinematic parts of a humanoid robot. Since not all joints of the robot are important for different planning phases, the RRT-based planner is able to adapt the number of DoF on the fly to improve the performance and the quality of the results. The runtime of the approach is evaluated in comparison to a standard RRT planner. Futhermore several extensions to the algorithm are investigated.

ICRA Conference 2008 Conference Paper

Object separation using active methods and multi-view representations

Kai Welke
Tamim Asfour
Rüdiger Dillmann

Daily life objects reveal natural similarities, which cannot be resolved with the perception of a single view. In this paper, we present an approach for object separation using active methods and multi-view object representations. By actively rotating an object, the coherence between controlled path, inner models, and percept is observed and used to reject implausible object hypotheses. Using the resulting object hypotheses, pose and object correspondence are determined. The proposed approach allows for the separation of different object candidates, which have similar views to the current percept. With the benefit of active methods the perceptual task can be solved using even coarse features, which facilitates a compact multi-view object representation. Furthermore, the approach is independent from a specific visual feature descriptor and thus suitable for multi-modal object recognition.

IROS Conference 2007 Conference Paper

Efficient motion planning for humanoid robots using lazy collision checking and enlarged robot models

Nikolaus Vahrenkamp
Tamim Asfour
Rüdiger Dillmann

Motion planning for humanoid robotic systems with many degrees of freedom is an important and still generally unsolved problem. To give the robot the ability of acting and navigating in complex environments, the motion planner has to find collision-free paths in a robust manner. The runtime of a planning algorithm is critical, since complex tasks require several planning steps where the collision detection and avoidance should be accomplished in reasonable time. In this paper we present an extension of standard sampling-based techniques using Rapidly Exploring Random Trees (RRT). We extend the free-bubble path validation algorithm from Quinlan, which can be used to guarantee the collision-free status of a C-space path between two samples. By using enlarged robot models it is possible to avoid costly distance calculations and therefore to speed up the planning process. We also present a combined approach based on lazy collision checking that brings together the advantages of fast sampling-based and exact path-validated algorithms. The proposed algorithms have been evaluated by experiments on a humanoid robot in a kitchen environment and by a comparison to a validation based on Quinlan's free bubbles approach.

ICRA Conference 2007 Conference Paper

Manipulation Planning Among Movable Obstacles

Mike Stilman
Jan-Ullrich Schamburek
James J. Kuffner
Tamim Asfour

This paper presents the resolve spatial constraints (RSC) algorithm for manipulation planning in a domain with movable obstacles. Empirically we show that our algorithm quickly generates plans for simulated articulated robots in a highly nonlinear search space of exponential dimension. RSC is a reverse-time search that samples future robot actions and constrains the space of prior object displacements. To optimize the efficiency of RSC, we identify methods for sampling object surfaces and generating connecting paths between grasps and placements. In addition to experimental analysis of RSC, this paper looks into object placements and task-space motion constraints among other unique features of the three dimensional manipulation planning domain.

IROS Conference 2007 Conference Paper

Stereo-based 6D object localization for grasping with humanoid robot systems

Pedram Azad
Tamim Asfour
Rüdiger Dillmann

Robust vision-based grasping is still a hard problem for humanoid robot systems. When being restricted to using the camera system built-in into the robot’s head for object localization, the scenarios get often very simplified in order to allow the robot to grasp autonomously. Within the computer vision community, many object recognition and localization systems exist, but in general, they are not tailored to the application on a humanoid robot. In particular, accurate 6D object localization in the camera coordinate system with respect to a 3D rigid model is crucial for a general framework for grasping. While many approaches try to avoid the use of stereo calibration, we will present a system that makes explicit use of the stereo camera system in order to achieve maximum depth accuracy. Our system can deal with textured objects as well as objects that can be segmented globally and are defined by their shape. Thus, it covers the cases of objects with complex texture and complex shape. Our work is directly linked to a grasping framework being implemented on the humanoid robot ARMAR and serves as its perception module for various grasping and manipulation experiments in a kitchen scenario.

ICRA Conference 2007 Conference Paper

Stereo-based Markerless Human Motion Capture for Humanoid Robot Systems

Pedram Azad
Ales Ude
Tamim Asfour
Rüdiger Dillmann

In this paper, we present an image-based markerless human motion capture system, intended for humanoid robot systems. The restrictions set by this ambitious goal are numerous. The input of the system is a sequence of stereo image pairs only, captured by cameras positioned at approximately eye distance. No artificial markers can be used to simplify the estimation problem. Furthermore, the complexity of all algorithms incorporated must be suitable for real-time application, which is maybe the biggest problem when considering the high dimensionality of the search space. Finally, the system must not depend on a static camera setup and has to find the initial configuration automatically. We present a system, which tackles these problems by combining multiple cues within a particle filter framework, allowing the system to recover from wrong estimations in a natural way. We make extensive use of the benefit of having a calibrated stereo setup. To reduce search space implicitly, we use the 3D positions of the hands and the head, computed by a separate hand and head tracker using a linear motion model for each entity to be tracked. With stereo input image sequences at a resolution of 320×240 pixels, the processing rate of our system is 15 Hz on a 3 GHz CPU. Experimental results documenting the performance of our system are available in form of several videos.

ICRA Conference 2007 Conference Paper

Toward an Unified Representation for Imitation of Human Motion on Humanoids

Pedram Azad
Tamim Asfour
Rüdiger Dillmann

In this paper, we present a framework for perception, visualization, reproduction and recognition of human motion. On the perception side, various human motion capture systems exist, all of them having in common to calculate a sequence of configuration vectors for the human model in the core of the system. These human models may be 2D or 3D kinematic models, or on a lower level, 2D or 3D positions of markers. However, for appropriate visualization in terms of a 3D animation, and for reproduction on an actual robot, the acquired motion must be mapped to the target 3D kinematic model. On the understanding side, various action and activity recognition systems exist, which assume input of different kinds. However, given human motion capture data in terms of a high-dimensional 3D kinematic model, it is possible to transform the configurations into the appropriate representation which is specific to the recognition module. We will propose a complete architecture, allowing the replacement of any perception, visualization, reproduction module, or target platform. In the core of our architecture, we define a reference 3D kinematic model, which we intend to become a common standard in the robotics community, to allow sharing different software modules and having common benchmarks.

ICRA Conference 2006 Conference Paper

An Integrated Approach to Inverse Kinematics and Path Planning for Redundant Manipulators

Dominik Bertram
James J. Kuffner
Rüdiger Dillmann
Tamim Asfour

We propose a novel solution to the problem of inverse kinematics for redundant robotic manipulators for the purposes of goal selection for path planning. We unify the calculation of the goal configuration with searching for a path in order to avoid the uncertainties inherent to selecting goal configurations which may be unreachable because they currently lie in components of the free configuration space disconnected from the initial configuration. We adopt workspace heuristic functions that implicitly define goal regions of the configuration space and guide the extension of rapidly-exploring random trees (RRTs), which are used to search for these regions. The algorithm has successfully been used to efficiently plan reaching and grasping motions for a humanoid robot equipped with redundant manipulator arms

IROS Conference 2006 Conference Paper

Combining Appearance-based and Model-based Methods for Real-Time Object Recognition and 6D Localization

Pedram Azad
Tamim Asfour
Rüdiger Dillmann

A general solution for image-based object recognition and localization is still a goal far away. Therefore, the only way to tackle the problem is to apply the suitable approach for each specific problem. The most common techniques can be classified into global appearance-based, model-based, or histogram-based approaches, and approaches based on local features. In this paper, we concentrate on recognition and full 6D localization of solid colored objects of any geometry for real-time application on a humanoid robot system. State-of-the-art model-based methods can only deal with object geometries which can be broken down into 3D lines and planes, and thus can be efficiently projected into the image plane, which is not the case for most objects in a realistic scenario. In contrast, appearance-based methods have the power to be applicable for any object geometry, but are rarely combined with full 6D localization of objects, which is required for any realistic application in the context of grasping with a humanoid robot. We present a system which combines the benefits of global appearance-based and model-based approaches, resulting in a system which can acquire object representations automatically given its 3D model, and can recognize and localize solid-colored objects in 6D in an arbitrary scene in real-time

IROS Conference 2006 Conference Paper

Integrated Grasp Planning and Visual Object Localization For a Humanoid Robot with Five-Fingered Hands

Antonio Morales
Tamim Asfour
Pedram Azad
Steffen Knoop
Rüdiger Dillmann

In this paper we present a framework for grasp planning with a humanoid robot arm and a five-fingered hand. The aim is to provide the humanoid robot with the ability of grasping objects that appear in a kitchen environment. Our approach is based on the use of an object model database that contains the description of all the objects that can appear in the robot workspace. This database is completed with two modules that make use of this object representation: an exhaustive offline grasp analysis system and a real-time stereo vision system. The offline grasp analysis system determines the best grasp for the objects by employing a simulation system, together with CAD models of the objects and the five-fingered hand. The results of this analysis are added to the object database using a description suited to the requirements of the grasp execution modules. A stereo camera system is used for a real-time object localization using a combination of appearance-based and model-based methods. The different components are integrated in a controller architecture to achieve manipulation task goals for the humanoid robot

IROS Conference 2004 Conference Paper

A modular and distributed embedded control architecture for humanoid robots

Duc Nguyen Ly
Kristian Regenstein
Tamim Asfour
Rüdiger Dillmann

In this paper we present a modular and distributed control architecture in order to achieve natural interaction and mobile manipulation task goals for a humanoid robot. We propose a hierarchically organized architecture with three levels and introduce the mapping of the functional features in this architecture into hardware and software modules. We also describe different functional features which have been realized and integrated into the whole control architecture.

IROS Conference 2004 Conference Paper

Programming by demonstration: dual-arm manipulation tasks for humanoid robots

Raoul Zöllner
Tamim Asfour
Rüdiger Dillmann

This paper deals with easy programming methods of dual-arm manipulation tasks for humanoid robots. Hereby a programming by demonstration system is used in order to observe, learn and generalize tasks performed by humans. A classification for dual-arm manipulations is introduced, enabling a segmentation of tasks into adequate subtasks. Further it is shown how the generated programs are mapped on and executed by a humanoid robot.

IROS Conference 2003 Conference Paper

Human-like motion of a humanoid robot arm based on a closed-form solution of the inverse kinematics problem

Tamim Asfour
Rüdiger Dillmann

Humanoid robotics is a new challenging field. To cooperate with human beings, humanoid robots not only have to feature human-like form and structure but, more importantly, they must possess human-like characteristics regarding motion, communication and intelligence. In this paper, we propose an algorithm for solving the inverse kinematics problem associated with the redundant robot arm of the humanoid robot ARMAR. The formulation of the problem is based on the decomposition of the workspace of the arm and on the analytical description of the redundancy of the arm. The solution obtained is characterized by its accuracy and low cost of computation. The algorithm is enhanced in order to generate human-like manipulation motions from object trajectories.

IROS Conference 2000 Conference Paper

Design of the TUAT/Karlsruhe humanoid hand

Naoki Fukaya
Shigeki Toyama
Tamim Asfour
Rüdiger Dillmann

The increasing demand for robotic applications in dynamic unstructured environments is motivating the need for dextrous end-effectors which can cope with the wide variety of tasks and objects encountered in these environments. The human hand is a very complex grasping tool that can handle objects of different sizes and shapes. Many research activities have been carried out to develop artificial robot hands with capabilities similar to the human hand. In this paper the mechanism and design of a new humanoid-type hand (called TUAT/Karlsruhe Humanoid Hand) with human-like manipulation abilities is discussed. The new hand is designed for the humanoid robot ARMAR which has to work autonomously or interactively in cooperation with humans and for an artificial lightweight arm for handicapped persons. The arm is developed as close as possible to the human arm and is driven by spherical ultrasonic motors. The ideal end-effector for such an artificial arm or a humanoid would be able to use the tools and objects that a person uses when working in the same environment. Therefore a new hand is designed for anatomical consistency with the human hand. This includes the number of fingers and the placement and motion of the thumb, the proportions of the link lengths and the shape of the palm. It can also perform most part of human grasping types. The TUAT/Karlsruhe Humanoid Hand possesses 20 DOF and is driven by one actuator which can be placed into or around the hand.

ICRA Conference 1999 Conference Paper

ARMAR: An Anthropomorphic Arm for Humanoid Service Robot

Karsten Berns
Tamim Asfour
Rüdiger Dillmann

Service robots which should perform human-like operations will penetrate into a great number of applications in the future. Requirements for this is high flexibility, autonomy and the ability to adapt to new situations. The paper describes a design concept and a prototype implementation of an autonomous mobile humanoid service robot, which should mainly support people in their daily life as a personal or an assistance robot. The state of the research is that the general concept is developed and two anthropomorphic arms are built up. In the article the sensor system and the control architecture of the anthropomorphic robot are described. To evaluate the performance and motion abilities of the anthropomorphic arm the human arm kinematics and properties are discussed.