Arrow Research search

Author name cluster

Sven Behnke

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

80 papers
2 author rows

Possible papers

80

AAAI Conference 2026 Conference Paper

Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation

  • Julian Spravil
  • Sebastian Houben
  • Sven Behnke

Cross-lingual, cross-task transfer is challenged by task-specific data scarcity, which becomes more severe as language support grows and is further amplified in vision-language models (VLMs). We investigate multilingual generalization in encoder-decoder transformer VLMs to enable zero-shot image captioning in languages encountered only in the translation task. In this setting, the encoder must learn to generate generalizable, task-aware latent vision representations to instruct the decoder via inserted cross-attention layers. To analyze scaling behavior, we train Florence-2 based and Gemma-2 based models (0.4B to 11.2B parameters) on a synthetic dataset using varying compute budgets. While all languages in the dataset have image-aligned translations, only a subset of them include image captions. Notably, we show that captioning can emerge using a language prefix, even when this language only appears in the translation task. We find that indirect learning of unseen task-language pairs adheres to scaling laws that are governed by the multilinguality of the model, model size, and seen training samples. Finally, we demonstrate that the scaling laws extend to downstream tasks, achieving competitive performance through fine-tuning in multimodal machine translation (Multi30K, CoMMuTE), lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO Karpathy).

PDF Details DOI

TMLR Journal 2026 Journal Article

TextOCVP: Object-Centric Video Prediction with Language Guidance

  • Angel Villar-Corrales
  • Gjergj Plepi
  • Sven Behnke

Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and interactions in order to predict future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotic environments. To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interactions while incorporating textual guidance, enabling accurate and controllable predictions. TextOCVP’s structured latent space offers a more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene configurations, as well as improved controllability and interpretability, enabling more precise and understandable predictions.

PDF Details

IROS Conference 2025 Conference Paper

DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models

  • Helin Cao
  • Sven Behnke

Perception systems play a crucial role in autonomous driving, incorporating multiple sensors and corresponding computer vision algorithms. 3D LiDAR sensors are widely used to capture sparse point clouds of the vehicle’s surroundings. However, such systems struggle to perceive occluded areas and gaps in the scene due to the sparsity of these point clouds and their lack of semantics. To address these challenges, Semantic Scene Completion (SSC) jointly predicts unobserved geometry and semantics in the scene given raw LiDAR measurements, aiming for a more complete scene representation. Building on promising results of diffusion models in image generation and super-resolution tasks, we propose their extension to SSC by implementing the noising and denoising diffusion processes in the point and semantic spaces individually. To control the generation, we employ semantic LiDAR point clouds as conditional input and design local and global regularization losses to stabilize the denoising process. We evaluate our approach on autonomous driving datasets, and it achieves state-of-the-art performance for SSC, surpassing most existing methods.

Details

ICML Conference 2025 Conference Paper

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

  • Angel Villar-Corrales
  • Sven Behnke

Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on videos and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available on our project website.

Details

ICRA Conference 2025 Conference Paper

Prompt-Responsive Object Retrieval with Memory-Augmented Student-Teacher Learning

  • Malte Mosbach
  • Sven Behnke

Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available at https://memory-student-teacher.github.io

Details

ICML Conference 2025 Conference Paper

SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels

  • Malte Mosbach
  • Jan Niklas Ewertz
  • Angel Villar-Corrales
  • Sven Behnke

Learning a latent dynamics model provides a task-agnostic representation of an agent’s understanding of its environment. Leveraging this knowledge for model-based reinforcement learning (RL) holds the potential to improve sample efficiency over model-free methods by learning from imagined rollouts. Furthermore, because the latent space serves as input to behavior models, the informative representations learned by the world model facilitate efficient learning of desired skills. Most existing methods rely on holistic representations of the environment’s state. In contrast, humans reason about objects and their interactions, predicting how actions will affect specific parts of their surroundings. Inspired by this, we propose Slot-Attention for Object-centric Latent Dynamics (SOLD), a novel model-based RL algorithm that learns object-centric dynamics models in an unsupervised manner from pixel inputs. We demonstrate that the structured latent space not only improves model interpretability but also provides a valuable input space for behavior models to reason over. Our results show that SOLD outperforms DreamerV3 and TD-MPC2 - state-of-the-art model-based RL algorithms - across a range of multi-object manipulation environments that require both relational reasoning and dexterous control. Videos and code are available at https: // slot-latent-dynamics. github. io.

Details

ICRA Conference 2024 Conference Paper

Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects

  • Malte Mosbach
  • Sven Behnke

Interactive grasping from clutter, akin to human dexterity, is one of the longest-standing problems in robot learning. Challenges stem from the intricacies of visual perception, the demand for precise motor skills, and the complex interplay between the two. In this work, we present Teacher-Augmented Policy Gradient (TAPG), a novel two-stage learning framework that synergizes reinforcement learning and policy distillation. After training a teacher policy to master the motor control based on object pose information, TAPG facilitates guided, yet adaptive, learning of a sensorimotor policy, based on object segmentation. We zero-shot transfer from simulation to a real robot by using Segment Anything Model for promptable object segmentation. Our trained policies adeptly grasp a wide variety of objects from cluttered scenarios in simulation and the real world based on human-understandable prompts. Furthermore, we show robust zero-shot transfer to novel objects. Videos of our experiments are available at https://maltemosbach.github.io/grasp_anything.

Details

IROS Conference 2024 Conference Paper

HortiBot: An Adaptive Multi-Arm System for Robotic Horticulture of Sweet Peppers

  • Christian Lenz
  • Rohit U. Menon
  • Michael Schreiber
  • Melvin Paul Jacob
  • Sven Behnke
  • Maren Bennewitz

Horticultural tasks such as pruning and selective harvesting are labor intensive and horticultural staff are hard to find. Automating these tasks is challenging due to the semi-structured greenhouse workspaces, changing environmental conditions such as lighting, dense plant growth with many occlusions, and the need for gentle manipulation of non-rigid plant organs. In this work, we present the three-armed system HortiBot, with two arms for manipulation and a third arm as an articulated head for active perception using stereo cameras. Its perception system detects not only peppers, but also peduncles and stems in real time, and performs online data association to build a world model of pepper plants. Collision-aware online trajectory generation allows all three arms to safely track their respective targets for observation, grasping, and cutting. We integrated perception and manipulation to perform selective harvesting of peppers and evaluated the system in lab experiments. Using active perception coupled with end-effector force torque sensing for compliant manipulation, HortiBot achieves high success rates in our indoor pepper plant mock-up.

Details

ECAI Conference 2024 Conference Paper

HyenaPixel: Global Image Context with Convolutions

  • Julian Spravil
  • Sebastian Houben
  • Sven Behnke

In computer vision, a larger effective receptive field (ERF) is associated with better performance. While attention natively supports global context, its quadratic complexity limits its applicability to tasks that benefit from high-resolution input. In this work, we extend Hyena, a convolution-based attention replacement, from causal sequences to bidirectional data and two-dimensional image space. We scale Hyena’s convolution kernels beyond the feature map size, up to 191×191, to maximize ERF while maintaining sub-quadratic complexity in the number of pixels. We integrate our two-dimensional Hyena, HyenaPixel, and bidirectional Hyena into the MetaFormer framework. For image categorization, HyenaPixel and bidirectional Hyena achieve a competitive ImageNet-1k top-1 accuracy of 84. 9% and 85. 2%, respectively, with no additional training data, while outperforming other convolutional and large-kernel networks. Combining HyenaPixel with attention further improves accuracy. We attribute the success of bidirectional Hyena to learning the data-dependent geometric arrangement of pixels without a fixed neighborhood definition. Experimental results on downstream tasks suggest that HyenaPixel with large filters and a fixed neighborhood leads to better localization performance.

Details

ICRA Conference 2024 Conference Paper

MOTPose: Multi-object 6D Pose Estimation for Dynamic Video Sequences using Attention-based Temporal Fusion

  • Arul Selvam Periyasamy
  • Sven Behnke

Cluttered bin-picking environments are challenging for pose estimation models. Despite the impressive progress enabled by deep learning, single-view RGB pose estimation models perform poorly in cluttered dynamic environments. Imbuing the rich temporal information contained in the video of scenes has the potential to enhance models’ ability to deal with the adverse effects of occlusion and the dynamic nature of the environments. Moreover, joint object detection and pose estimation models are better suited to leverage the co-dependent nature of the tasks for improving the accuracy of both tasks. To this end, we propose attention-based temporal fusion for multi-object 6D pose estimation that accumulates information across multiple frames of a video sequence. Our MOTPose method takes a sequence of images as input and performs joint object detection and pose estimation for all objects in one forward pass. It learns to aggregate both object embeddings and object parameters over multiple time steps using cross-attention-based fusion modules. We evaluate our method on the physically-realistic cluttered bin-picking dataset SynPick and the YCB-Video dataset and demonstrate improved pose estimation accuracy as well as better object detection accuracy.

Details

ICRA Conference 2024 Conference Paper

SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net

  • Helin Cao
  • Sven Behnke

We introduce SLCF-Net, a novel approach for the Semantic Scene Completion (SSC) task that sequentially fuses LiDAR and camera data. It jointly estimates missing geometry and semantics in a scene from sequences of RGB images and sparse LiDAR measurements. The images are semantically segmented by a pre-trained 2D U-Net and a dense depth prior is estimated from a depth-conditioned pipeline fueled by Depth Anything. To associate the 2D image features with the 3D scene volume, we introduce Gaussian-decay Depth-prior Projection (GDP). This module projects the 2D features into the 3D volume along the line of sight with a Gaussian-decay function, centered around the depth prior. Volumetric semantics is computed by a 3D U-Net. We propagate the hidden 3D U-Net state using the sensor motion and design a novel loss to ensure temporal consistency. We evaluate our approach on the SemanticKITTI dataset and compare it with leading SSC approaches. The SLCF-Net excels in all SSC metrics and shows great temporal consistency.

Details

IROS Conference 2023 Conference Paper

Attention-Based VR Facial Animation with Visual Mouth Camera Guidance for Immersive Telepresence Avatars

  • Andre Rochow
  • Max Schwarz
  • Sven Behnke

Facial animation in virtual reality environments is essential for applications that necessitate clear visibility of the user's face and the ability to convey emotional signals. In our scenario, we animate the face of an operator who controls a robotic Avatar system. The use of facial animation is particularly valuable when the perception of interacting with a specific individual, rather than just a robot, is intended. Purely keypoint-driven animation approaches struggle with the complexity of facial movements. We present a hybrid method that uses both keypoints and direct visual guidance from a mouth camera. Our method generalizes to unseen operators and requires only a quick enrolment step with capture of two short videos. Multiple source images are selected with the intention to cover different facial expressions. Given a mouth camera frame from the HMD, we dynamically construct the target keypoints and apply an attention mechanism to determine the importance of each source image. To resolve keypoint ambiguities and animate a broader range of mouth expressions, we propose to inject visual mouth camera information into the latent space. We enable training on large-scale speaking head datasets by simulating the mouth camera input with its perspective differences and facial deformations. Our method outperforms a baseline in quality, capability, and temporal consistency. In addition, we highlight how the facial animation contributed to our victory at the ANA Avatar XPRIZE Finals.

Details

IROS Conference 2023 Conference Paper

Dynamic Hybrid Locomotion and Jumping for Wheeled-Legged Quadrupeds

  • Mojtaba Hosseini
  • Diego Rodriguez
  • Sven Behnke

Hybrid wheeled-legged quadrupeds have the potential to navigate challenging terrain with agility and speed and over long distances. However, obstacles can impede their progress by requiring the robots to either slow down to step over obstacles or modify their path to circumvent the obstacles. We propose a motion optimization framework for quadruped robots that incorporates non-steerable wheels and dynamic jumps, enabling them to perform hybrid wheeled-legged locomotion while overcoming obstacles without slowing down. Our approach involves a model predictive controller that uses a time-varying rigid body dynamics model of the robot, including legs and wheels, to track dynamic motions such as jumping. We also introduce a method for driving with minimal leg swings to reduce energy consumption by sparing the effort involved in lifting the wheels. Our method was tested successfully on the wheeled Mini Cheetah and the Unitree AlienGo robots. Further videos and results are available at https://www.ais.uni-bonn.de/∼hosseini/iros2023

Details

ICRA Conference 2023 Conference Paper

External Camera-Based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors

  • Simon Bultmann
  • Raphael Memmesheimer
  • Sven Behnke

We present an approach for estimating a mobile robot's pose w. r. t. the allocentric coordinates of a network of static cameras using multi-view RGB images. The images are processed online, locally on smart edge sensors by deep neural networks to detect the robot and estimate 2D keypoints defined at distinctive positions of the 3D robot model. Robot keypoint detections are synchronized and fused on a central backend, where the robot's pose is estimated via multi-view minimization of reprojection errors. Through the pose estimation from external cameras, the robot's localization can be initialized in an allocentric map from a completely unknown state (kidnapped robot problem) and robustly tracked over time. We conduct a series of experiments evaluating the accuracy and robustness of the camera-based pose estimation compared to the robot's internal navigation stack, showing that our camera-based method achieves pose errors below 3 cm and 1° and does not drift over time, as the robot is localized allocentrically. With the robot's pose precisely estimated, its observations can be fused into the allocentric scene model. We show a real-world application, where observations from mobile robot and static smart edge sensors are fused to collaboratively build a 3D semantic map of a ~240 m 2 indoor environment.

Details

IROS Conference 2023 Conference Paper

Quadrupedal Footstep Planning Using Learned Motion Models of a Black-Box Controller

  • Ilyass Taouil
  • Giulio Turrisi
  • Daniel Schleich
  • Victor Barasuol
  • Claudio Semini
  • Sven Behnke

Legged robots are increasingly entering new domains and applications, including search and rescue, inspection, and logistics. However, for such a systems to be valuable in real-world scenarios, they must be able to autonomously and robustly navigate irregular terrains. In many cases, robots that are sold on the market do not provide such abilities, being able to perform only blind locomotion. Furthermore, their controller cannot be easily modified by the end-user, requiring a new and time-consuming control synthesis. In this work, we present a fast local motion planning pipeline that extends the capabilities of a black-box walking controller that is only able to track high-level reference velocities. More precisely, we learn a set of motion models for such a controller that maps high-level velocity commands to Center of Mass (CoM) and footstep motions. We then integrate these models with a variant of the $A$ * algorithm to plan the CoM trajectory, footstep sequences, and corresponding high-level velocity commands based on visual information, allowing the quadruped to safely traverse irregular terrains at demand.

Details

ICRA Conference 2022 Conference Paper

Abstract Flow for Temporal Semantic Segmentation on the Permutohedral Lattice

  • Peer Schütt
  • Radu Alexandru Rosu
  • Sven Behnke

Semantic segmentation is a core ability required by autonomous agents, as being able to distinguish which parts of the scene belong to which object class is crucial for navigation and interaction with the environment. Approaches which use only one time-step of data cannot distinguish between moving objects nor can they benefit from temporal integration. In this work, we extend a backbone LatticeNet to process temporal point cloud data. Additionally, we take inspiration from optical flow methods and propose a new module called Abstract Flow which allows the network to match parts of the scene with similar abstract features and gather the information temporally. We obtain state-of-the-art results on the SemanticKITTI dataset that contains LiDAR scans from real urban environments. We share the PyTorch implementation of TemporalLatticeNet at https://github.com/AIS-Bonn/temporal_latticenet.

Details

IROS Conference 2022 Conference Paper

Predictive Angular Potential Field-based Obstacle Avoidance for Dynamic UAV Flights

  • Daniel Schleich
  • Sven Behnke

In recent years, unmanned aerial vehicles (UAVs) are used for numerous inspection and video capture tasks. Manually controlling UAVs in the vicinity of obstacles is challenging, however, and poses a high risk of collisions. Even for autonomous flight, global navigation planning might be too slow to react to newly perceived obstacles. Disturbances such as wind might lead to deviations from the planned trajectories. In this work, we present a fast predictive obstacle avoidance method that does not depend on higher-level localization or mapping and maintains the dynamic flight capabilities of UAVs. It directly operates on LiDAR range images in real time and adjusts the current flight direction by computing angular potential fields within the range image. The velocity magnitude is subsequently determined based on a trajectory prediction and time-to-contact estimation. Our method is evaluated using Hardware-in-the-Loop simulations. It keeps the UAV at a safe distance to obstacles, while allowing higher flight velocities than previous reactive obstacle avoidance methods that directly operate on sensor data.

Details

ICRA Conference 2022 Conference Paper

Real-Robot Deep Reinforcement Learning: Improving Trajectory Tracking of Flexible-Joint Manipulator with Reference Correction

  • Dmytro Pavlichenko
  • Sven Behnke

Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correction actions and use of a model-free off-policy learning method. In addition, an informed policy initialization is proposed, where the agent is pre-trained in a learned simulation. We test our approach on the 7 DOF manipulator of a Baxter robot. We demonstrate that the proposed method is capable of consistent learning across multiple runs when applied directly on the real robot. Our method yields a policy which significantly improves the trajectory tracking accuracy in comparison to the vendor-provided controller, generalizing to an unseen payload.

Details

IROS Conference 2022 Conference Paper

VR Facial Animation for Immersive Telepresence Avatars

  • Andre Rochow
  • Max Schwarz
  • Michael Schreiber
  • Sven Behnke

VR Facial Animation is necessary in applications requiring clear view of the face, even though a VR headset is worn. In our case, we aim to animate the face of an operator who is controlling our robotic avatar system. We propose a real-time capable pipeline with very fast adaptation for specific operators. In a quick enrollment step, we capture a sequence of source images from the operator without the VR headset which contain all the important operator-specific appearance information. During inference, we then use the operator keypoint information extracted from a mouth camera and two eye cameras to estimate the target expression and head pose, to which we map the appearance of a source still image. In order to enhance the mouth expression accuracy, we dynamically select an auxiliary expression frame from the captured sequence. This selection is done by learning to transform the current mouth keypoints into the source camera space, where the alignment can be determined accurately. We, furthermore, demonstrate an eye tracking pipeline that can be trained in less than a minute, a time efficient way to train the whole pipeline given a dataset that includes only complete faces, show exemplary results generated by our method, and discuss performance at the ANA Avatar XPRIZE semifinals.

Details

ICRA Conference 2021 Conference Paper

DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning

  • Diego Rodriguez
  • Sven Behnke

Bipedal walking is one of the most difficult but exciting challenges in robotics. The difficulties arise from the complexity of high-dimensional dynamics, sensing and actuation limitations combined with real-time and computational constraints. Deep Reinforcement Learning (DRL) holds the promise to address these issues by fully exploiting the robot dynamics with minimal craftsmanship. In this paper, we propose a novel DRL approach that enables an agent to learn omnidirectional locomotion for humanoid (bipedal) robots. Notably, the locomotion behaviors are accomplished by a single control policy (a single neural network). We achieve this by introducing a new curriculum learning method that gradually increases the task difficulty by scheduling target velocities. In addition, our method does not require reference motions which facilities its application to robots with different kinematics, and reduces the overall complexity. Finally, different strategies for sim-to-real transfer are presented which allow us to transfer the learned policy to a real humanoid robot.

Details

IROS Conference 2021 Conference Paper

Mapless Humanoid Navigation Using Learned Latent Dynamics

  • André Brandenburger
  • Diego Rodriguez
  • Sven Behnke

In this paper, we propose a novel Deep Reinforcement Learning approach to address the mapless navigation problem, in which the locomotion actions of a humanoid robot are taken online based on the knowledge encoded in learned models. Planning happens by generating open-loop trajectories in a learned latent space that captures the dynamics of the environment. Our planner considers visual (RGB images) and non-visual observations (e. g. , attitude estimations). This confers the agent upon awareness not only of the scenario, but also of its own state. In addition, we incorporate a termination likelihood predictor model as an auxiliary loss function of the control policy, which enables the agent to anticipate terminal states of success and failure. In this manner, the sample efficiency of the approach for episodic tasks is increased. Our model is evaluated on the NimbRo-OP2X humanoid robot that navigates in scenes avoiding collisions efficiently in simulation and with the real hardware.

Details

IROS Conference 2021 Conference Paper

NimbRo Avatar: Interactive Immersive Telepresence with Force-Feedback Telemanipulation

  • Max Schwarz
  • Christian Lenz
  • Andre Rochow
  • Michael Schreiber
  • Sven Behnke

Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typical indoor scenarios, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR HMD worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying a synthesized image of the operator with facial animation, which enables direct interaction with remote persons. We evaluate our system successfully both in a user study with untrained operators as well as a longer and more complex integrated mission. We discuss lessons learned from the trials and possible improvements.

Details

IROS Conference 2021 Conference Paper

Real-time Multi-Adaptive-Resolution-Surfel 6D LiDAR Odometry using Continuous-time Trajectory Optimization

  • Jan Quenzel
  • Sven Behnke

Simultaneous Localization and Mapping (SLAM) is an essential capability for autonomous robots, but due to high data rates of 3D LiDARs real-time SLAM is challenging. We propose a real-time method for 6D LiDAR odometry. Our approach combines a continuous-time B-Spline trajectory representation with a Gaussian Mixture Model (GMM) formulation to jointly align local multi-resolution surfel maps. Sparse voxel grids and permutohedral lattices ensure fast access to map surfels, and an adaptive resolution selection scheme effectively speeds up registration. A thorough experimental evaluation shows the performance of our approach on multiple datasets and during real-robot experiments.

Details

ICRA Conference 2021 Conference Paper

Search-based Planning of Dynamic MAV Trajectories Using Local Multiresolution State Lattices

  • Daniel Schleich
  • Sven Behnke

Search-based methods that use motion primitives can incorporate the system’s dynamics into the planning and thus generate dynamically feasible MAV trajectories that are globally optimal. However, searching high-dimensional state lattices is computationally expensive. Local multiresolution is a commonly used method to accelerate spatial path planning. While paths within the vicinity of the robot are represented at high resolution, the representation gets coarser for more distant parts. In this work, we apply the concept of local multiresolution to high-dimensional state lattices that include velocities and accelerations. Experiments show that our proposed approach significantly reduces planning times. Thus, it increases the applicability to large dynamic environments, where frequent replanning is necessary.

Details

ICRA Conference 2020 Conference Paper

Beyond Photometric Consistency: Gradient-based Dissimilarity for Improving Visual Odometry and Stereo Matching

  • Jan Quenzel
  • Radu Alexandru Rosu
  • Thomas Läbe
  • Cyrill Stachniss
  • Sven Behnke

Pose estimation and map building are central ingredients of autonomous robots and typically rely on the registration of sensor data. In this paper, we investigate a new metric for registering images that builds upon on the idea of the photometric error. Our approach combines a gradient orientation-based metric with a magnitude-dependent scaling term. We integrate both into stereo estimation as well as visual odometry systems and show clear benefits for typical disparity and direct image registration tasks when using our proposed metric. Our experimental evaluation indicate that our metric leads to more robust and more accurate estimates of the scene depth as well as camera trajectory. Thus, the metric improves camera pose estimation and in turn the mapping capabilities of mobile robots. We believe that a series of existing visual odometry and visual SLAM systems can benefit from the findings reported in this paper.

Details

IROS Conference 2020 Conference Paper

Category-Level 3D Non-Rigid Registration from Single-View RGB Images

  • Diego Rodriguez
  • Florian Huber
  • Sven Behnke

In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field (typically used for transferring knowledge between instances, e. g. , grasping skills) that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As result of the registration, the observed model is reconstructed. Because our method does not need depth information, it can register objects that are typically hard to perceive with RGB-D sensors, e. g. with transparent or shiny surfaces. Even without depth data, our approach outperforms the Coherent Point Drift (CPD) registration method for the evaluated object categories.

Details

ICRA Conference 2020 Conference Paper

Fast Whole-Body Motion Control of Humanoid Robots with Inertia Constraints

  • Grzegorz Ficht
  • Sven Behnke

We introduce a new, analytical method for generating whole-body motions for humanoid robots, which approximate the desired Composite Rigid Body (CRB) inertia. Our approach uses a reduced five mass model, where four of the masses are attributed to the limbs and one is used for the trunk. This compact formulation allows for finding an analytical solution that combines the kinematics with mass distribution and inertial properties of a humanoid robot. The positioning of the masses in Cartesian space is then directly used to obtain joint angles with relations based on simple geometry. Motions are achieved through the time evolution of poses generated through the desired foot positioning and CRB inertia properties. As a result, we achieve short computation times in the order of tens of microseconds. This makes the method suited for applications with limited computation resources, or leaving them to be spent on higher-layer tasks such as model predictive control. The approach is evaluated by performing a dynamic kicking motion with an igus ® Humanoid Open Platform robot.

Details

ICRA Conference 2020 Conference Paper

Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

  • Max Schwarz
  • Sven Behnke

Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduced by the camera sensors are simulated. Our pipeline can be run online during training of a deep neural network, yielding applications in life-long learning and in iterative render-and-compare approaches. We demonstrate the usability by learning semantic segmentation on the challenging YCB-Video dataset without actually using any training frames, where our method achieves performance comparable to a conventionally trained model. Additionally, we show successful application in a real-world regrasping system.

Details

IROS Conference 2019 Conference Paper

A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

  • Patrick Stotko
  • Stefan Krumpen
  • Max Schwarz
  • Christian Lenz
  • Sven Behnke
  • Reinhard Klein
  • Michael Weinmann

Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multiclient telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for e. g. head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to first- person exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot, complementing traditional input devices for teleoperation. We provide a proof-of-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization. Furthermore, we show its benefits over purely video-based teleoperation in a user study revealing a higher degree of situation awareness and a more precise navigation in challenging environments.

Details

ICRA Conference 2019 Conference Paper

Detection and Tracking of Small Objects in Sparse 3D Laser Range Data

  • Jan Razlaw
  • Jan Quenzel
  • Sven Behnke

Detection and tracking of dynamic objects is a key feature for autonomous behavior in a continuously changing environment. With the increasing popularity and capability of micro aerial vehicles (MAVs) efficient algorithms have to be utilized to enable multi object tracking on limited hardware and data provided by lightweight sensors. We present a novel segmentation approach based on a combination of median filters and an efficient pipeline for detection and tracking of small objects within sparse point clouds generated by a Velodyne VLP-16 sensor. We achieve real-time performance on a single core of our MAV hardware by exploiting the inherent structure of the data. Our approach is evaluated on simulated and real scans of in- and outdoor environments, obtaining results comparable to the state of the art. Additionally, we provide an application for filtering the dynamic and mapping the static part of the data, generating further insights into the performance of the pipeline on unlabeled data.

Details

IROS Conference 2019 Conference Paper

Directional TSDF: Modeling Surface Orientation for Coherent Meshes

  • Malte Splietker
  • Sven Behnke

Real-time 3D reconstruction from RGB-D sensor data plays an important role in many robotic applications, such as object modeling and mapping. The popular method of fusing depth information into a truncated signed distance function (TSDF) and applying the marching cubes algorithm for mesh extraction has severe issues with thin structures: not only does it lead to loss of accuracy, but it can generate completely wrong surfaces. To address this, we propose the directional TSDF—a novel representation that stores opposite surfaces separate from each other. The marching cubes algorithm is modified accordingly to retrieve a coherent mesh representation. We further increase the accuracy by using surface gradient-based ray casting for fusing new measurements. We show that our method outperforms state-of-the-art TSDF reconstruction algorithms in mesh accuracy.

Details

IROS Conference 2019 Conference Paper

Fast Time-optimal Avoidance of Moving Obstacles for High-Speed MAV Flight

  • Marius Beul
  • Sven Behnke

In this work, we propose a method to efficiently compute smooth, time-optimal trajectories for micro aerial vehicles (MAVs) evading a moving obstacle. Our approach first computes an n-dimensional trajectory from the start- to an arbitrary target state including position, velocity and acceleration. It respects input- and state-constraints and is thus dynamically feasible. The trajectory is then efficiently checked for collisions, exploiting the piecewise polynomial formulation. If collisions occur, viastates are inserted into the trajectory to circumvent the obstacle and still maintain time-optimality. These viastates are described by position, velocity, and acceleration. The evaluation shows that the computational demands of the proposed method are minimal such that obstacle avoidance can begin within few milliseconds. Optimality of generated trajectories, combined with the ability for frequent online re-planning from non-hover initial conditions, make the approach well suited for evasion of suddenly perceived obstacles during fast flight.

Details

ICRA Conference 2019 Conference Paper

Search-based 3D Planning and Trajectory Optimization for Safe Micro Aerial Vehicle Flight Under Sensor Visibility Constraints

  • Matthias Nieuwenhuisen
  • Sven Behnke

Safe navigation of Micro Aerial Vehicles (MAVs) requires not only obstacle-free flight paths according to a static environment map, but also the perception of and reaction to previously unknown and dynamic objects. This implies that the onboard sensors cover the current flight direction. Due to the limited payload of MAVs, full sensor coverage of the environment has to be traded off with flight time. Thus, often only a part of the environment is covered. We present a combined allocentric complete planning and trajectory optimization approach taking these sensor visibility constraints into account. The optimized trajectories yield flight paths within the apex angle of a Velodyne Puck Lite 3D laser scanner enabling low-level collision avoidance to perceive obstacles in the flight direction. Furthermore, the optimized trajectories take the flight dynamics into account and contain the velocities and accelerations along the path. We evaluate our approach with a DJI Matrice 600 MAV and in simulation employing hardware-in-the-loop.

Details

ICRA Conference 2019 Conference Paper

Towards Learning Abstract Representations for Locomotion Planning in High-dimensional State Spaces

  • Tobias Klamt
  • Sven Behnke

Ground robots which are able to navigate a variety of terrains are needed in many domains. One of the key aspects is the capability to adapt to the ground structure, which can be realized through movable body parts coming along with additional degrees of freedom (DoF). However, planning respective locomotion is challenging since suitable representations result in large state spaces. Employing an additional abstract representation-which is coarser, lower-dimensional, and semantically enriched-can support the planning. While a desired robot representation and action set of such an abstract representation can be easily defined, the cost function requires large tuning efforts. We propose a method to represent the cost function as a CNN. Training of the network is done on generated artificial data, while it generalizes well to the abstraction of real world scenes. We further apply our method to the problem of search-based planning of hybrid driving-stepping locomotion. The abstract representation is used as a powerful informed heuristic which accelerates planning by multiple orders of magnitude.

Details

ICRA Conference 2018 Conference Paper

Efficient Continuous-Time SLAM for 3D Lidar-Based Online Mapping

  • David Droeschel
  • Sven Behnke

Modern 3D laser-range scanners have a high data rate, making online simultaneous localization and mapping (SLAM) computationally challenging. Recursive state estimation techniques are efficient but commit to a state estimate immediately after a new scan is made, which may lead to misalignments of measurements. We present a 3D SLAM approach that allows for refining alignments during online mapping. Our method is based on efficient local mapping and a hierarchical optimization back-end. Measurements of a 3D laser scanner are aggregated in local multiresolution maps by means of surfel-based registration. The local maps are used in a multi-level graph for allocentric mapping and localization. In order to incorporate corrections when refining the alignment, the individual 3D scans in the local map are modeled as a sub-graph and graph optimization is performed to account for drift and misalignments in the local maps. Furthermore, in each sub-graph, a continuous-time representation of the sensor trajectory allows to correct measurements between scan poses. We evaluate our approach in multiple experiments by showing qualitative results. Furthermore, we quantify the map quality by an entropy-based measure.

Details

ICRA Conference 2018 Conference Paper

Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing

  • Max Schwarz
  • Christian Lenz
  • Germán Martín García
  • Seongyong Koo
  • Arul Selvam Periyasamy
  • Michael Schreiber
  • Sven Behnke

Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a custom turntable capture system and transfer learning. It produces high-quality item segments, on which grasp poses are found. A planning component coordinates manipulation actions between two robot arms, minimizing execution time. The system has been demonstrated successfully at ARC, where our team reached second places in both the picking task and the final stow-and-pick task. We also evaluate individual components.

Details

IROS Conference 2018 Conference Paper

Fused Angles and the Deficiencies of Euler Angles

  • Philipp Allgeuer
  • Sven Behnke

Just like the well-established Euler angles representation, fused angles are a convenient parameterisation for rotations in three-dimensional Euclidean space. They were developed in the context of balancing bodies, most specifically walking bipedal robots, but have since found wider application due to their useful properties. A comparative analysis between fused angles and Euler angles is presented in this paper, delineating the specific differences between the two representations that make fused angles more suitable for representing orientations in balance-related scenarios. Aspects of comparison include the locations of the singularities, the associated parameter sensitivities, the level of mutual independence of the parameters, and the axisymmetry of the parameters.

Details

IROS Conference 2018 Conference Paper

Keyframe-Based Photometric Online Calibration and Color Correction

  • Jan Quenzel
  • Jannis Horn
  • Sebastian Houben
  • Sven Behnke

Finding the parameters of a vignetting function for a camera currently involves the acquisition of several images in a given scene under very controlled lighting conditions, a cumbersome and error-prone task where the end result can only be confirmed visually. Many computer vision algorithms assume photoconsistency, constant intensity between scene points in different images, and tend to perform poorly if this assumption is violated. We present a real-time online vignetting and response calibration with additional exposure estimation for global-shutter color cameras. Our method does not require uniformly illuminated surfaces, known texture or specific geometry. The only assumptions are that the camera is moving, the illumination is static and reflections are Lambertian. Our method estimates the camera view poses by sparse visual SLAM and models the vignetting function by a small number of thin plate splines (TPS) together with a sixth-order polynomial to provide a dense estimation of attenuation from sparsely sampled scene points. The camera response function (CRF) is jointly modeled by a TPS and a Gamma curve. We evaluate our approach on synthetic datasets and in real-world scenarios with reference data from a Structure-from-Motion (SfM) system. We show clear visual improvement on textured meshes without the need for extensive meshing algorithms. A useful calibration is obtained from a few keyframes which makes an on-the-fly deployment conceivable.

Details

ICRA Conference 2018 Conference Paper

Planning Hybrid Driving-Stepping Locomotion on Multiple Levels of Abstraction

  • Tobias Klamt
  • Sven Behnke

Navigating in search and rescue environments is challenging, since a variety of terrains has to be considered. Hybrid driving-stepping locomotion, as provided by our robot Momaro, is a promising approach. Similar to other locomotion methods, it incorporates many degrees of freedom - offering high flexibility but making planning computationally expensive for larger environments. We propose a navigation planning method, which unifies different levels of representation in a single planner. In the vicinity of the robot, it provides plans with a fine resolution and a high robot state dimensionality. With increasing distance from the robot, plans become coarser and the robot state dimensionality decreases. We compensate this loss of information by enriching coarser representations with additional semantics. Experiments show that the proposed planner provides plans for large, challenging scenarios in feasible time.

Details

IROS Conference 2018 Conference Paper

Robust 6D Object Pose Estimation in Cluttered Scenes Using Semantic Segmentation and Pose Regression Networks

  • Arul Selvam Periyasamy
  • Max Schwarz
  • Sven Behnke

Object pose estimation is a crucial prerequisite for robots to perform autonomous manipulation in clutter. Real-world bin-picking settings such as warehouses present additional challenges, e. g. , new objects are added constantly. Most of the existing object pose estimation methods assume that 3D models of the objects is available beforehand. We present a pipeline that requires minimal human intervention and circumvents the reliance on the availability of 3D models by a fast data acquisition method and a synthetic data generation procedure. This work builds on previous work on semantic segmentation of cluttered bin-picking scenes to isolate individual objects in clutter. An additional network is trained on synthetic scenes to estimate object poses from a cropped object-centered encoding extracted from the segmentation results. The proposed method is evaluated on a synthetic validation dataset and cluttered realworld scenes.

Details

IROS Conference 2018 Conference Paper

Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur-Like Robot

  • Tobias Klamt
  • Diego Rodriguez
  • Max Schwarz
  • Christian Lenz
  • Dmytro Pavlichenko
  • David Droeschel
  • Sven Behnke

Mobile manipulation tasks are one of the key challenges in the field of search and rescue (SAR) robotics requiring robots with flexible locomotion and manipulation abilities. Since the tasks are mostly unknown in advance, the robot has to adapt to a wide variety of terrains and workspaces during a mission. The centaur-like robot Centauro has a hybrid legged-wheeled base and an anthropomorphic upper body to carry out complex tasks in environments too dangerous for humans. Due to its high number of degrees of freedom, controlling the robot with direct teleoperation approaches is challenging and exhausting. Supervised autonomy approaches are promising to increase quality and speed of control while keeping the flexibility to solve unknown tasks. We developed a set of operator assistance functionalities with different levels of autonomy to control the robot for challenging locomotion and manipulation tasks. The integrated system was evaluated in disaster response scenarios and showed promising performance.

Details

ICRA Conference 2018 Conference Paper

Transferring Grasping Skills to Novel Instances by Latent Space Non-Rigid Registration

  • Diego Rodriguez
  • Corbin Cogswell
  • Seongyong Koo
  • Sven Behnke

Robots acting in open environments need to be able to handle novel objects. Based on the observation that objects within a category are often similar in their shapes and usage, we propose an approach for transferring grasping skills from known instances to novel instances of an object category. Correspondences between the instances are established by means of a non-rigid registration method that combines the Coherent Point Drift approach with subspace methods. The known object instances are modeled using a canonical shape and a transformation which deforms it to match the instance shape. The principle axes of variation of these deformations define a low-dimensional latent space. New instances can be generated through interpolation and extrapolation in this shape space. For inferring the shape parameters of an unknown instance, an energy function expressed in terms of the latent variables is minimized. Due to the class-level knowledge of the object, our method is able to complete novel shapes from partial views. Control poses for generating grasping motions are transferred efficiently to novel instances by the estimated non-rigid transformation.

Details

IROS Conference 2017 Conference Paper

Anytime hybrid driving-stepping locomotion planning

  • Tobias Klamt
  • Sven Behnke

Hybrid driving-stepping locomotion is an effective approach for navigating in a variety of environments. Long, sufficiently even distances can be quickly covered by driving while obstacles can be overcome by stepping. Our quadruped robot Momaro, with steerable pairs of wheels located at the end of each of its compliant legs, allows such locomotion. Planning respective paths attracted only little attention so far. We propose a navigation planning method which generates hybrid locomotion paths. The planner chooses driving mode whenever possible and takes into account the detailed robot footprint. If steps are required, the planner includes those. To accelerate planning, steps are planned first as abstract manoeuvres and are expanded afterwards into detailed motion sequences. Our method ensures at all times that the robot stays stable. Experiments show that the proposed planner is capable of providing paths in feasible time, even for challenging terrain.

Details

IROS Conference 2017 Conference Paper

Efficient stochastic multicriteria arm trajectory optimization

  • Dmytro Pavlichenko
  • Sven Behnke

Performing manipulation with robotic arms requires a method for planning trajectories that takes multiple factors into account: collisions, joint limits, orientation constraints, torques, and duration of a trajectory. We present an approach to efficiently optimize arm trajectories with respect to multiple criteria. Our work extends Stochastic Trajectory Optimization for Motion Planning (STOMP). We optimize trajectory duration by including velocity into the optimization. We propose an efficient cost function with normalized components, which allows prioritizing components depending on user-specified requirements. Optimization is done in two stages: first with a partial cost function and in the second stage with full costs. We compare our method to state-of-the art methods. In addition, we perform experiments on real robots: centaur-like robot Momaro and an industrial manipulator.

Details

ICRA Conference 2017 Conference Paper

NimbRo picking: Versatile part handling for warehouse automation

  • Max Schwarz
  • Anton Milan
  • Christian Lenz
  • Aura Munoz
  • Arul Selvam Periyasamy
  • Michael Schreiber
  • Sebastian Schüller
  • Sven Behnke

Part handling in warehouse automation is challenging if a large variety of items must be accommodated and items are stored in unordered piles. To foster research in this domain, Amazon holds picking challenges. We present our system which achieved second and third place in the Amazon Picking Challenge 2016 tasks. The challenge required participants to pick a list of items from a shelf or to stow items into the shelf. Using two deep-learning approaches for object detection and semantic segmentation and one item model registration method, our system localizes the requested item. Manipulation occurs using suction on points determined heuristically or from 6D item model registration. Parametrized motion primitives are chained to generate motions. We present a full-system evaluation during the APC 2016 and component-level evaluations of the perception system on an annotated dataset.

Details

IROS Conference 2017 Conference Paper

Online depth calibration for RGB-D cameras using visual SLAM

  • Jan Quenzel
  • Radu Alexandru Rosu
  • Sebastian Houben
  • Sven Behnke

Modern consumer RGB-D cameras are affordable and provide dense depth estimates at high frame rates. Hence, they are popular for building dense environment representations. Yet, the sensors often do not provide accurate depth estimates since the factory calibration exhibits a static deformation. We present a novel approach to online depth calibration that uses a visual SLAM system as reference for the measured depth. A sparse map is generated and the visual information is used to correct the static deformation of the measured depth while missing data is extrapolated using a small number of thin plate splines (TPS). The corrected depth can then be used to improve the accuracy of the sparse RGB-D map and the 3D environment reconstruction. As more data becomes available, the depth calibration is updated on the fly. Our method does not rely on a planar geometry like walls or a one-to-one-pixel correspondence between color and depth camera. Our approach is evaluated in real-world scenarios and against ground truth data. Comparison against two popular self-calibration methods is performed. Furthermore, we show clear visual improvement on aggregated point clouds with our method.

Details

IROS Conference 2017 Conference Paper

Online visual robot tracking and identification using deep LSTM networks

  • Hafez Farazi
  • Sven Behnke

Collaborative robots working on a common task are necessary for many applications. One of the challenges for achieving collaboration in a team of robots is mutual tracking and identification. We present a novel pipeline for online vision-based detection, tracking and identification of robots with a known and identical appearance. Our method runs in realtime on the limited hardware of the observer robot. Unlike previous works addressing robot tracking and identification, we use a data-driven approach based on recurrent neural networks to learn relations between sequential inputs and outputs. We formulate the data association problem as multiple classification problems. A deep LSTM network was trained on a simulated dataset and fine-tuned on small set of real data. Experiments on two challenging datasets, one synthetic and one real, which include long-term occlusions, show promising results.

Details

IROS Conference 2016 Conference Paper

Efficient multi-camera visual-inertial SLAM for micro aerial vehicles

  • Sebastian Houben
  • Jan Quenzel
  • Nicola Krombach
  • Sven Behnke

Visual SLAM is an area of vivid research and bears countless applications for moving robots. In particular, micro aerial vehicles benefit from visual sensors due to their low weight. Their motion is, however, often faster and more complex than that of ground-based robots which is why systems with multiple cameras are currently evaluated and deployed. This, in turn, drives the computational demand for visual SLAM algorithms.

Details

ICRA Conference 2016 Conference Paper

Focused online visual-motor coordination for a dual-arm robot manipulator

  • Seongyong Koo
  • Sven Behnke

Coordination between visual sensors and robot manipulators is necessary for successful manipulation. This paper proposes a novel visual-motor coordination method that performs online parameter estimation of an RGB-D camera mounted in a robot head without any external markers. Through self-observation of a dual-arm robot manipulator, the method updates parameters to reduce the discrepancy between observed point cloud data and 3D mesh models of the current robot con guration. With the estimated parameters at each time step, visual data is adjusted to the focused workspace of the 14DOF dual-arm robot manipulator. The online and realtime algorithm was developed by using a GPU-based particle filtering method. Experimental results show that our method outperforms state-of-the-art offline registration methods in terms of accuracy and computation time. We also analyzed the dependence of the results on prior parameters to demonstrate the online capability of our method.

Details

ICRA Conference 2016 Conference Paper

Hybrid driving-stepping locomotion with the wheeled-legged robot Momaro

  • Max Schwarz
  • Tobias Rodehutskors
  • Michael Schreiber
  • Sven Behnke

Locomotion in uneven terrain is important for a wide range of robotic applications, including Search&Rescue operations. Our mobile manipulation robot Momaro features a unique locomotion design consisting of four legs ending in pairs of steerable wheels, allowing the robot to omnidirectionally drive on sufficiently even terrain, step over obstacles, and also to overcome height differences by climbing. We demonstrate the feasibility and usefulness of this design on the example of the DARPA Robotics Challenge, where our team NimbRo Rescue solved seven out of eight tasks in only 34 minutes. We also introduce a method for semi-autonomous execution of weight-shifting and stepping actions based on a 2D heightmap generated from 3D laser data.

Details

IROS Conference 2016 Conference Paper

Local multiresolution trajectory optimization for micro aerial vehicles employing continuous curvature transitions

  • Matthias Nieuwenhuisen
  • Sven Behnke

Complex indoor and outdoor missions for autonomous micro aerial vehicles (MAV) require fast generation of collision-free paths in 3D space. Often not all obstacles in an environment are known prior to the mission execution. Consequently, the ability for replanning during a flight is key for success. Our approach locally optimizes trajectories of grid-based path planning. It preserves obstacle-freeness of the path and ensures smoothness with continuous curvature transition segments. Fast optimization and frequent reoptimization is made possible by means of local multiresolution time discretization. With our extensions, high dimensional flight trajectories incorporating velocities and accelerations can be planned with a time discretization of 100 Hz within the prediction horizon of the underlying controller.

Details

IROS Conference 2015 Conference Paper

Fused angles: A representation of body orientation for balance

  • Philipp Allgeuer
  • Sven Behnke

The parameterisation of rotations in three dimensional Euclidean space is an area of applied mathematics that has long been studied, dating back to the original works of Euler in the 18 th century. As such, many ways of parameterising a rotation have been developed over the years. Motivated by the task of representing the orientation of a balancing body, the fused angles parameterisation is developed and introduced in this paper. This novel representation is carefully defined both mathematically and geometrically, and thoroughly investigated in terms of the properties it possesses, and how it relates to other existing representations. A second intermediate representation, tilt angles, is also introduced as a natural consequence thereof.

Details

IROS Conference 2015 Conference Paper

Gradient-driven online learning of bipedal push recovery

  • Marcell Missura
  • Sven Behnke

Bipedal walking is a complex and dynamic whole-body motion with balance constraints. Due to the inherently unstable inverted pendulum-like dynamics of walking, the design of robust walking controllers proved to be particularly challenging. While a controller could potentially be learned with a robot in the loop, the destructive nature of losing balance and the impracticality of a high number of repetitions render most existing learning methods unsuitable for an online learning setting with real hardware. We propose a model-driven learning method that enables a humanoid robot to quickly learn how to maintain its balance. We bootstrap the learning process with a central pattern generator for stepping motions that abstracts from the complexity of the walking motion and simplifies the problem setting to the learning of a small number of leg swing amplitude parameters. A simple physical model that represents the dominant dynamics of bipedal walking estimates an approximate gradient and suggests how to modify the swing amplitude to restore balance. In experiments with a real robot, we show that only a few failed steps are sufficient for our biped to learn strong push recovery skills in the sagittal direction.

Details

IROS Conference 2015 Conference Paper

Real-time object detection, localization and verification for fast robotic depalletizing

  • Dirk Holz
  • Angeliki Topalidou-Kyniazopoulou
  • Jörg Stückler
  • Sven Behnke

Depalletizing is a challenging task for manipulation robots. Key to successful application are not only robustness of the approach, but also achievable cycle times in order to keep up with the rest of the process. In this paper, we propose a system for depalletizing and a complete pipeline for detecting and localizing objects as well as verifying that the found object does not deviate from the known object model, e. g. , if it is not the object to pick. In order to achieve high robustness (e. g. , with respect to different lighting conditions) and generality with respect to the objects to pick, our approach is based on multi-resolution surfel models. All components (both software and hardware) allow operation at high frame rates and, thus, allow for low cycle times. In experiments, we demonstrate depalletizing of automotive and other prefabricated parts with both high reliability (w. r. t. success rates) and efficiency (w. r. t. low cycle times).

Details

ICRA Conference 2015 Conference Paper

RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features

  • Max Schwarz
  • Hannes Schulz
  • Sven Behnke

Object recognition and pose estimation from RGB-D images are important tasks for manipulation robots which can be learned from examples. Creating and annotating datasets for learning is expensive, however. We address this problem with transfer learning from deep convolutional neural networks (CNN) that are pre-trained for image categorization and provide a rich, semantically meaningful feature set. We incorporate depth information, which the CNN was not trained with, by rendering objects from a canonical perspective and colorizing the depth channel according to distance from the object center. We evaluate our approach on the Washington RGB-D Objects dataset, where we find that the generated feature set naturally separates classes and instances well and retains pose manifolds. We outperform state-of-the-art on a number of subtasks and show that our approach can yield superior results when only little training data is available.

Details

ICRA Conference 2014 Conference Paper

Bayesian exploration and interactive demonstration in continuous state MAXQ-learning

  • Kathrin Gräve
  • Sven Behnke

Deploying robots for service tasks requires learning algorithms that scale to the combinatorial complexity of our daily environment. Inspired by the way humans decompose complex tasks, hierarchical methods for robot learning have attracted significant interest. In this paper, we apply the MAXQ method for hierarchical reinforcement learning to continuous state spaces. By using Gaussian Process Regression for MAXQ value function decomposition, we obtain probabilistic estimates of primitive and completion values for every subtask within the MAXQ hierarchy. From these, we recursively compute probabilistic estimates of state-action values. Based on the expected deviation of these estimates, we devise a Bayesian exploration strategy that balances optimization of expected values and risk from exploring unknown actions. To further reduce risk and to accelerate learning, we complement MAXQ with learning from demonstrations in an interactive way. In every situation and subtask, the system may ask for a demonstration if there is not enough knowledge available to determine a safe action for exploration. We demonstrate the ability of the proposed system to efficiently learn solutions to complex tasks on a box stacking scenario.

Details

ICRA Conference 2014 Conference Paper

Efficient deformable registration of multi-resolution surfel maps for object manipulation skill transfer

  • Jörg Stückler
  • Sven Behnke

Endowing mobile manipulation robots with skills to use objects and tools often involves the programming or training on specific object instances. To apply this knowledge to novel instances from the same class of objects, a robot requires generalization capabilities for control as well as perception. In this paper, we propose an efficient approach to deformable registration of RGB-D images that enables robots to transfer skills between object instances. Our method provides a dense deformation field between the current image and an object model which allows for estimating local rigid transformations on the object's surface. Since we define grasp and motion strategies as poses and trajectories with respect to the object models, these strategies can be transferred to novel instances through local transformations derived from the deformation field. In experiments, we demonstrate the accuracy and runtime efficiency of our registration method. We also report on the use of our skill transfer approach in a public demonstration.

Details

ICRA Conference 2014 Conference Paper

Learning depth-sensitive conditional random fields for semantic segmentation of RGB-D images

  • Andreas C. Müller 0001
  • Sven Behnke

We present a structured learning approach to semantic annotation of RGB-D images. Our method learns to reason about spatial relations of objects and fuses low-level class predictions to a consistent interpretation of a scene. Our model incorporates color, depth and 3D scene features, on which an energy function is learned to directly optimize object class prediction using the loss-based maximum-margin principle of structural support vector machines. We evaluate our approach on the NYU V2 dataset of indoor scenes, a challenging dataset covering a wide variety of scene layouts and object classes. We hard-code much less information about the scene layout into our model then previous approaches, and instead learn object relations directly from the data. We find that our conditional random field approach improves upon previous work, setting a new state-of-the-art for the dataset.

Details

ICRA Conference 2014 Conference Paper

Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner

  • David Droeschel
  • Jörg Stückler
  • Sven Behnke

Micro aerial vehicles (MAV) pose a challenge in designing sensory systems and algorithms due to their size and weight constraints and limited computing power. We present an efficient 3D multi-resolution map that we use to aggregate measurements from a lightweight continuously rotating laser scanner. We estimate the robot's motion by means of visual odometry and scan registration, aligning consecutive 3D scans with an incrementally built map. By using local multi-resolution, we gain computational efficiency by having a high resolution in the near vicinity of the robot and a lower resolution with increasing distance from the robot, which correlates with the sensor's characteristics in relative distance accuracy and measurement density. Compared to uniform grids, local multi-resolution leads to the use of fewer grid cells without loosing information and consequently results in lower computational costs. We efficiently and accurately register new 3D scans with the map in order to estimate the motion of the MAV and update the map in-flight. In experiments, we demonstrate superior accuracy and efficiency of our registration approach compared to state-of-the-art methods such as GICP. Our approach builds an accurate 3D obstacle map and estimates the vehicle's trajectory in real-time.

Details

JMLR Journal 2014 Journal Article

pystruct - Learning Structured Prediction in Python

  • Andreas C. Müller
  • Sven Behnke

Structured prediction methods have become a central tool for many machine learning applications. While more and more algorithms are developed, only very few implementations are available. pystruct aims at providing a general purpose implementation of standard structured prediction methods, both for practitioners and as a baseline for researchers. It is written in Python and adapts paradigms and types from the scientific Python community for seamless integration with other projects. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2014. ( edit, beta )

PDF Details

IJCAI Conference 2013 Conference Paper

Hierarchical Object Discovery and Dense Modelling from Motion Cues in RGB-D Video

  • Jörg Stückler
  • Sven Behnke

In this paper, we propose a novel method for object discovery and dense modelling in RGB-D image sequences using motion cues. We develop our method as a building block for active object perception, such that robots can learn about the environment through perceiving the effects of actions. Our approach simultaneously segments rigid-body motion within key views, and discovers objects and hierarchical relations between object parts. The poses of the key views are optimized in a graph of spatial relations to recover the rigid-body motion trajectories of the camera with respect to the objects. In experiments, we demonstrate that our approach finds moving objects, aligns partial views on the objects, and retrieves hierarchical relations between the objects.

PDF Details DOI

IROS Conference 2013 Conference Paper

Learning sequential tasks interactively from demonstrations and own experience

  • Kathrin Gräve
  • Sven Behnke

Deploying robots to our day-to-day life requires them to have the ability to learn from their environment in order to acquire new task knowledge and to flexibly adapt existing skills to various situations. For typical real-world tasks, it is not sufficient to endow robots with a set of primitive actions. Rather, they need to learn how to sequence these in order to achieve a desired effect on their environment. In this paper, we propose an intuitive learning method for a robot to acquire sequences of motions by combining learning from human demonstrations and reinforcement learning. In every situation, our approach treats both ways of learning as alternative control flows to optimally exploit their strengths without inheriting their shortcomings. Using a Gaussian Process approximation of the state-action sequence value function, our approach generalizes values observed from demonstrated and autonomously generated action sequences to unknown inputs. This approximation is based on a kernel we designed to account for different representations of tasks and action sequences as well as inputs of variable length. From the expected deviation of value estimates, we devise a greedy exploration policy following a Bayesian optimization criterion that quickly converges learning to promising action sequences while protecting the robot from sequences with unpredictable outcome. We demonstrate the ability of our approach to efficiently learn appropriate action sequences in various situations on a manipulation task involving stacked boxes.

Details

ICRA Conference 2013 Conference Paper

Mobile bin picking with an anthropomorphic service robot

  • Matthias Nieuwenhuisen
  • David Droeschel
  • Dirk Holz
  • Jörg Stückler
  • Alexander Berner
  • Jun Li 0042
  • Reinhard Klein
  • Sven Behnke

Grasping individual objects from an unordered pile in a box has been investigated in static scenarios so far. In this paper, we demonstrate bin picking with an anthropomorphic mobile robot. To this end, we extend global navigation techniques by precise local alignment with a transport box. Objects are detected in range images using a shape primitive-based approach. Our approach learns object models from single scans and employs active perception to cope with severe occlusions. Grasps and arm motions are planned in an efficient local multiresolution height map. All components are integrated and evaluated in a bin picking and part delivery task.

Details

IROS Conference 2012 Conference Paper

Incremental action recognition and generalizing motion generation based on goal-directed features

  • Kathrin Gräve
  • Sven Behnke

The ability to recognize human actions is a fundamental problem in many areas of robotics research concerned with human-robot interaction or learning from human demonstration. In this paper, we present a new integrated approach to identifying and recognizing actions in human movement sequences and their reproduction in unknown situations. We propose a set of task-space features to construct probabilistic models of action classes. Based on this representation, we suggest a combined segmentation and classification algorithm which processes data non-greedily using an incremental lookahead to reliably locate transitions between actions. In a programming by demonstration scenario, our action models afford the generalization and reproduction of learned movements to previously unseen situations. To evaluate the performance of our approach, we consider typical manipulation tasks in a table top setting. In a sequence of human demonstrations, our approach successfully extracts and recognizes actions from different classes and subsequently generalizes them to unknown situations.

Details

AAAI Conference 2012 Conference Paper

Model Learning and Real-Time Tracking Using Multi-Resolution Surfel Maps

  • Jörg Stückler
  • Sven Behnke

For interaction with its environment, a robot is required to learn models of objects and to perceive these models in the livestreams from its sensors. In this paper, we propose a novel approach to model learning and real-time tracking. We extract multi-resolution 3D shape and texture representations from RGB-D images at high frame-rates. An efficient variant of the iterative closest points algorithm allows for registering maps in real-time on a CPU. Our approach learns full-view models of objects in a probabilistic optimization framework in which we find the best alignment between multiple views. Finally, we track the pose of the camera with respect to the learned model by registering the current sensor view to the model. We evaluate our approach on RGB-D benchmarks and demonstrate its accuracy, efficiency, and robustness in model learning and tracking. We also report on the successful public demonstration of our approach in a mobile manipulation task.

PDF Details

IROS Conference 2012 Conference Paper

Semantic mapping using object-class segmentation of RGB-D images

  • Jörg Stückler
  • Nenad Biresev
  • Sven Behnke

For task planning and execution in unstructured environments, a robot needs the ability to recognize and localize relevant objects. When this information is made persistent in a semantic map, it can be used, e. g. , to communicate with humans. In this paper, we propose a novel approach to learning such maps. Our approach registers measurements of RGB-D cameras by means of simultaneous localization and mapping. We employ random decision forests to segment object classes in images and exploit dense depth measurements to obtain scale-invariance. Our object recognition method integrates shape and texture seamlessly. The probabilistic segmentation from multiple views is filtered in a voxel-based 3D map using a Bayesian framework. We report on the quality of our object-class segmentation method and demonstrate the benefits in accuracy when fusing multiple views in a semantic map.

Details

ICRA Conference 2011 Conference Paper

Efficient kinodynamic trajectory generation for wheeled robots

  • Marcell Missura
  • Sven Behnke

Planning dynamic motion is computationally demanding and thus can hardly be done in real-time onboard robots. In this paper, we present an analytic approximation to predict the dynamic state of wheeled robots with non-holonomic constraints, given a start state and a sequence of piecewise constant controls. Our approximations are accurate and fast to calculate. They can be used to replace numerical integrators in kinodynamic planning algorithms. The predictions are differentiable and allow us to utilize gradient descent methods to solve the inverse dynamics as well and generate trajectories connecting arbitrary points in state space.

Details

ICRA Conference 2011 Conference Paper

Interest point detection in depth images through scale-space surface analysis

  • Jörg Stückler
  • Sven Behnke

Many perception problems in robotics such as object recognition, scene understanding, and mapping are tackled using scale-invariant interest points extracted from intensity images. Since interest points describe only local portions of objects and scenes, they offer robustness to clutter, occlusions, and intra-class variation. In this paper, we present an efficient approximate algorithm to extract surface normal interest points (SNIPs) in corners and blob-like surface regions from depth images. The interest points are detected on characteristic scales that indicate their spatial extent. Our method is able to cope with irregularly sampled, noisy measurements which are typical to depth imaging devices. It also offers a trade-off between computational speed and accuracy which allows our approach to be applicable in a wide range of problem sets. We evaluate our approach on depth images of basic geometric shapes, more complex objects, and indoor scenes.

Details

ICRA Conference 2011 Conference Paper

Towards joint attention for a domestic service robot - person awareness and gesture recognition using Time-of-Flight cameras

  • David Droeschel
  • Jörg Stückler
  • Dirk Holz
  • Sven Behnke

Joint attention between a human user and a robot is essential for effective human-robot interaction. In this work, we propose an approach to person awareness and to the perception of showing and pointing gestures for a domestic service robot. In contrast to previous work, we do not require the person to be at a predefined position, but instead actively approach and orient towards the communication partner. For perceiving showing and pointing gestures and for estimating the pointing direction a Time-of-Flight camera is used. Estimated pointing directions and shown objects are matched to objects in the robot's environment. Both the perception of showing and pointing gestures as well as the accurary of estimated pointing directions have been evaluated in a set of different experiments. The results show that both gestures are adequatly perceived by the robot. Furthermore, our system achieves a higher accuracy in estimating the pointing direction than is reported in the literature for a stereo-based system. In addition, the overall system has been successfully tested in two international RoboCup@Home competitions and the 2010 ICRA Mobile Manipulation Challenge.

Details

IROS Conference 2010 Conference Paper

Combining depth and color cues for scale- and viewpoint-invariant object segmentation and recognition using Random Forests

  • Jörg Stückler
  • Sven Behnke

In this paper we present an approach to object segmentation and recognition that combines depth and color cues. We fuse information from color images with depth from a Time-of-Flight (ToF) camera to improve recognition performance under scale and viewpoint changes. Firstly, we use depth and local surface orientation extracted from the ToF image to normalize color and depth image features with regard to scale and viewpoint. Secondly, we incorporate local 3D shape features into the classifier. The use of a Random Forest classifier facilitates the seamless combination of depth and texture features. It also provides image segmentation through pixel-wise classification. We demonstrate our approach on a labeled dataset of seven object categories in table-top scenes and compare it with a vision-only approach.

Details

ICRA Conference 2010 Conference Paper

Improving indoor navigation of autonomous robots by an explicit representation of doors

  • Matthias Nieuwenhuisen
  • Jörg Stückler
  • Sven Behnke

In the last decades, tremendous progress has been made in the field of autonomous indoor navigation for mobile robots. However, these approaches assume the structural part of the environment to be completely static. In practice, movable parts of scenes, e. g. doors, frequently violate this assumption which leads to poor performance. Also, mobile manipulation capabilities can only be utilized, if the robot knows about the movability of objects. In this paper, we address an important part of these problems by the explicit representation of doors as door leaves and joints. We propose to augment standard approaches to navigation like 2D occupancy grid mapping and Monte-Carlo-Localization. Our algorithm detects doors during mapping and represents their movability adequately in the map. During localization, the state of doors is estimated from measurements while it is simultaneously used to improve localization robustness and accuracy. In experimental results we demonstrate superior performance of our method compared to a state-of-the-art approach to localization.

Details

IROS Conference 2010 Conference Paper

Multi-frequency Phase Unwrapping for Time-of-Flight cameras

  • David Droeschel
  • Dirk Holz
  • Sven Behnke

Time-of-Flight (ToF) cameras gain depth information by emitting amplitude-modulated near-infrared light and measuring the phase shift between the emitted and the reflected signal. The phase shift is proportional to the object's distance modulo the wavelength of the modulation frequency. This results in a distance ambiguity. Distances larger than the wavelength are wrapped into the sensor's non-ambiguity range and cause spurious distance measurements. We apply Phase Unwrapping to reconstruct these wrapped measurements. Our approach is based on a probabilistic graphical model. We use loopy belief propagation to detect and infer the position of wrapped measurements. Besides depth discontinuities, our method utilizes multiple modulation frequencies to identify wrapped measurements. In experiments, we show that wrapped measurements are identified and corrected, even in situations where the scene shows steep slopes in the depth measurements.

Details

ICRA Conference 2010 Conference Paper

Sancta simplicitas - on the efficiency and achievable results of SLAM using ICP-based incremental registration

  • Dirk Holz
  • Sven Behnke

This paper presents an efficient combination of algorithms for SLAM in dynamic environments. The overall approach is based on range image registration using the ICP algorithm. Different extensions to this algorithm are used to incrementally construct point models of the robot's workspace. A simple heuristic allows for determining which points in a newly acquired range image are already contained in the point model and for adding only those points that provide new information. Furthermore, the means for dealing with environment dynamics are presented which allow for continuously conducting SLAM and updating the point model according to changes in a dynamic environment. The achievable results of the overall approach are compared to Rao-Blackwellized Particle Filters as a state-of-the-art solution to the SLAM problem and evaluated using a recently published benchmark by Burgard et al. (2009).

Details

ICRA Conference 2010 Conference Paper

Using Time-of-Flight cameras with active gaze control for 3D collision avoidance

  • David Droeschel
  • Dirk Holz
  • Jörg Stückler
  • Sven Behnke

We propose a 3D obstacle avoidance method for mobile robots. Besides the robot's 2D laser range finder, a Time-of-Flight camera is used to perceive obstacles that are not in the scan plane of the laser range finder. Existing approaches that employ Time-of-Flight cameras suffer from the limited field-of-view of the sensor. To overcome this issue, we mount the camera on the head of our anthropomorphic robot Dynamaid. This allows to change the gaze direction through the robot's pan-tilt neck and its torso yaw joint. The proposed obstacle detection method is robust against kinematic inaccuracies and noise in the range measurements. The gaze controller takes motion blur effects into account and controls the gaze depending on the robot's motion and the obstacles in its vicinity. In experiments, we demonstrate that our approach enables the robot to avoid obstacles that the laser range finder can not perceive. We also compare our active gaze control strategy with a fixed gaze orientation.

Details

ICRA Conference 2009 Conference Paper

Utilizing reflection properties of surfaces to improve mobile robot localization

  • Maren Bennewitz
  • Cyrill Stachniss
  • Sven Behnke
  • Wolfram Burgard

A main difficulty that arises in the context of probabilistic localization is the design of an appropriate observation model, i. e. , determining the likelihood of a sensor measurement given the pose of the robot and a map of the environment. Many successful approaches to localization rely on data provided by range sensors, e. g. , laser range scanners. When using such data one normally has to deal with erroneous maximum-range readings that occur due to poor-reflecting surfaces. In general, these readings cannot be distinguished from readings obtained when no obstacle is within the measurement range of the sensor. Therefore, existing localization techniques treat these readings alike in the observation model. In this paper, we present a novel approach that explicitly considers the reflection properties of surfaces and thus the expectation of valid range measurements. In addition to the expected range measurement, we compute the probability of reflectance for a beam given the relative pose of the robot to the obstacle taking into account the angle of incidence of the beam. We estimate the reflection properties of surfaces using data collected with a mobile robot equipped with a laser range scanner. As we demonstrate in experiments carried out with a real robot, our technique leads to significantly improved localization results compared to a state-of-the-art observation model.

Details

ICRA Conference 2008 Conference Paper

How to learn accurate grid maps with a humanoid

  • Cyrill Stachniss
  • Maren Bennewitz
  • Giorgio Grisetti
  • Sven Behnke
  • Wolfram Burgard

Humanoids have recently become a popular research platform in the robotics community. Such robots offer various fields for new applications. However, they have several drawbacks compared to wheeled vehicles such as stability problems, limited payload capabilities, violation of the flat world assumption, and they typically provide only very rough odometry information, if at all. In this paper, we investigate the problem of learning accurate grid maps with humanoid robots. We present techniques to deal with some of the above-mentioned difficulties. We describe how an existing approach to the simultaneous localization and mapping (SLAM) problem can be adapted to robustly learn accurate maps with a humanoid equipped with a laser range finder. We present an experiment in which our mapping system builds a highly accurate map with a size of around 20 m by 20 m using data acquired with a humanoid in our office environment containing two loops. The resulting maps have a similar accuracy as maps built with a wheeled robot.

Details

ICRA Conference 2008 Conference Paper

Orthogonal wall correction for visual motion estimation

  • Jörg Stückler
  • Sven Behnke

A good motion model is a prerequisite for many approaches to simultaneous localization and mapping. Without an absolute reference, it is however difficult to prevent drift when estimating motion. To prevent orientation drift, our approach exploits typical features of indoor environments: Straight walls that are parallel or orthogonal to each other. Our idea is to detect walls in monocular depth measurements and to correct odometry obtained from matching successive images and from inertial measurements, such that the observed walls are aligned with the main orientation estimated from the map that is being built. The experimental results indicate that orientation drift can be prevented and orientation uncertainty can be reduced greatly when applying the proposed orthogonal wall correction. This can make the difference between reliable mapping and failure.

Details

IROS Conference 2006 Conference Paper

Instability Detection and Fall Avoidance for a Humanoid using Attitude Sensors and Reflexes

  • Reimund Renner
  • Sven Behnke

Humanoid robots are inherently unstable because their center of mass is high, compared to the support polygon's size. Bipedal walking currently works well only under controlled conditions with limited external disturbances. In less controlled dynamic environments, such as RoboCup soccer fields, external disturbances might be large. While some disturbances might be too large to prevent a fall, some disturbances can be dealt with by specific rescue behaviors. This paper proposes a method to detect instabilities that occur during omnidirectional walking. We model the readings of attitude sensors using sinusoids. The model takes the gait target vector into account. We estimate model parameters from a gait test sequence and detect deviations of the actual sensor readings from the model later on. These deviations are aggregated to an instability indicator that triggers one of two reflexes, based on indicator strength. For small instabilities the robot is slowing down, but continues walking. For stronger instabilities the robot stops and is brought into a stable posture with a low center of mass. Walking continues as soon as the instability disappears. We extensively evaluated our approach in simulation by disturbing the robot with a variety of impulses. The results indicate that our method is very effective. For smaller disturbances, the probability of a fall could be reduced to zero. Most of the medium-sized disturbances could also be rejected. For the evaluation with the real robot, we used a walking against a wall with different speeds and at various angles. Here the results show a similar outcome to the ones in the simulations

Details

ICRA Conference 2006 Conference Paper

Online Trajectory Generation for Omnidirectional Biped Walking

  • Sven Behnke

This paper describes the online generation of trajectories for omnidirectional walking on two legs. The gait can be parameterized using walking direction, walking speed, and rotational speed. Our approach has a low computational complexity and can be implemented on small onboard computers. We tested the proposed approach using our humanoid robot Jupp. The competitions in the RoboCup soccer domain showed that omnidirectional walking has advantages when acting in dynamic environments

Details

IROS Conference 2005 Conference Paper

Integrating vision and speech for conversations with multiple persons

  • Maren Bennewitz
  • Felix Faber
  • Dominik Joho
  • Michael Schreiber
  • Sven Behnke

An essential capability for a robot designed to interact with humans is to show attention to the people in its surroundings. To enable a robot to involve multiple persons into interaction requires the maintenance of an accurate belief about the people in the environment. In this paper, we use a probabilistic technique to update the knowledge of the robot based on sensory input. In this way, the robot is able to reason about the uncertainty in its belief about people in the vicinity and is able to shift its attention between different persons. Even people who are not the primary conversational partners are included into the interaction. In practical experiments with a humanoid robot, we demonstrate the effectiveness of our approach.

Details
v2026.09.13