Michael Beetz

AAAI Conference 2026 Conference Paper

GRIM: Task-Oriented Grasping with Conditioning on Generative Examples

• Shailesh Shailesh
• Alok Raj
• Nayan Kumar
• Priya Shukla
• Andrew Melnik
• Michael Beetz
• Gora Chand Nandi

Task-Oriented Grasping (TOG) presents a significant challenge, requiring a nuanced understanding of task semantics, object affordances, and the functional constraints dictating how an object should be grasped for a specific task. To address these challenges, we introduce GRIM (Grasp Re-alignment via Iterative Matching), a novel training-free framework for task-oriented grasping. Initially, a coarse alignment strategy is developed using a combination of geometric cues and principal component analysis (PCA)-reduced DINO features for similarity scoring. Subsequently, the full grasp pose associated with the retrieved memory instance is transferred to the aligned scene object and further refined against a set of task-agnostic, geometrically stable grasps generated for the scene object, prioritizing task compatibility. In contrast to existing learning-based methods, GRIM demonstrates strong generalization capabilities, achieving robust performance with only a small number of conditioning examples.

JAAMAS Journal 2026 Journal Article

Structured Reactive Controllers

• Michael Beetz

Abstract Service robots, such as autonomous office couriers or robot tourguides, must be both reliable and efficient. This requires them to flexibly interleave their tasks, exploit opportunities, quickly plan their course of action, and, if necessary, revise their intended activities. In this paper, we show how structured reactive controllers (SRCs) satisfy these requirements. The novel feature of SRCs is that they employ and reason about plans that specify and synchronize concurrent percept-driven behavior. Powerful control abstractions enable SRCs to integrate physical action, perception, planning, and communication in a uniform framework and to apply fast but imperfect computational methods without sacrificing reliability and flexibility. Concurrent plans are represented in a transparent and modular form so that automatic planning processes can reason about the plans and revise them. We present experiments in which SRCs are used to control two autonomous mobile robots. In one of them an SRC controlled the course of action of a museum tourguide robot that has operated for thirteen days, more than ninetyfour hours, completed 620 tours, and presented 2668 exhibits.

IROS Conference 2025 Conference Paper

Generating Actionable Robot Knowledge Bases by Combining 3D Scene Graphs with Robot Ontologies

• Giang Nguyen
• Mihai Pomarlan
• Sascha Jongebloed
• Nils Leusmann
• Minh Nhat Vu
• Michael Beetz

In robotics, the effective integration of environ-mental data into actionable knowledge remains a significant challenge due to the variety and incompatibility of data formats commonly used in scene descriptions, such as MJCF, URDF, and SDF. This paper presents a novel approach that addresses these challenges by developing a unified scene graph model that standardizes these varied formats into the Universal Scene Description (USD) format. This standardization facilitates the integration of these scene graphs with robot ontologies through semantic reporting, enabling the translation of complex environmental data into actionable knowledge essential for cognitive robotic control. We evaluated our approach by converting procedural 3D environments into USD format, which is then annotated semantically and translated into a knowledge graph to effectively answer competency questions, demonstrating its utility for real-time robotic decision-making. Additionally, we developed a web-based visualization tool to support the semantic mapping process, providing users with an intuitive interface to manage the 3D environment.

ICRA Conference 2025 Conference Paper

Shadow Program Inversion with Differentiable Planning: A Framework for Unified Robot Program Parameter and Trajectory Optimization

• Benjamin Alt
• Claudius Kienle
• Darko Katic
• Rainer Jäkel
• Michael Beetz

This paper presents Shadow Program Inversion with Differentiable Planning (SPI-DP), a novel first-order optimizer capable of optimizing robot programs with respect to both high-level task objectives and motion-level constraints. To that end, we introduce Differentiable Gaussian Process Motion Planning for N-DoF Manipulators (dGPMP2-ND), a differentiable collision-free motion planner for serial N-DoF kinematics, and integrate it into an iterative, gradient-based optimization approach for generic, parameterized robot program representations. SPI-DP allows first-order optimization of planned trajectories and program parameters with respect to objectives such as cycle time or smoothness subject to e. g. collision constraints, while enabling humans to understand, modify or even certify the optimized programs. We provide a comprehensive evaluation on two practical household and industrial applications.

ICRA Conference 2025 Conference Paper

Towards Autonomous Verification: Integrating Cognitive AI and Semantic Digital Twins in Medical Robotics

• Patrick Mania
• Michael Neumann
• Franklin Kenghagho Kenfack
• Michael Beetz

In medical laboratory environments, where pre-cision and safety are critical, the deployment of autonomous robots requires not only accurate object manipulation but also the ability to verify task success to comply with regulatory requirements. This paper introduces a novel imagination-enabled perception framework that integrates cognitive AI with semantic digital twins to allow medical robots to sim-ulate task outcomes, compare them with real-world results, and autonomously verify the success of their actions. Our approach addresses challenges related to handling small and transparent objects commonly found in sterility testing kits and other related consumables. By enhancing the RoboKudo perception system with parthood-based reasoning, we enable more accurate task verification through focused attention on object subparts. Experiments show that our system significantly improves performance compared to traditional object-centric methods, increasing accuracy in complex environments without the need for extensive retraining. This work demonstrates a novel concept in making robotic systems more adaptable and reliable for critical tasks in medical laboratories.

ICRA Conference 2024 Conference Paper

An Open and Flexible Robot Perception Framework for Mobile Manipulation Tasks

• Patrick Mania
• Simon Stelter
• Gayane Kazhoyan
• Michael Beetz

Over the last years, powerful methods for solving specific perception problems such as object detection, pose estimation or scene understanding have been developed. While performing mobile manipulation actions, a robot’s perception framework needs to execute a series of these methods in a specific sequence each time it receives a new perception task. Generating proficient combinations of vision methods to solve individual perception tasks remains a challenge, as the combination depends on the requirements of the task and the capabilities of the robot’s hardware. In this paper, we propose RoboKudo, an open-source knowledge-enabled perception framework that leverages the strengths of the Unstructured Information Management (UIM) principle and the flexibility of Behavior Trees to model task-specific perception processes. The framework can combine state-of-the-art computer vision methods to satisfy the requirements of each perception task and scales to different robot platforms. The generality and effectiveness of the framework are evaluated in real world experiments where it solves various perception tasks in the context of mobile manipulation actions in a household domain. Code and additional material are available at https://robokudo. ai. uni-bremen. de/rkop.

ICRA Conference 2024 Conference Paper

Perception through Cognitive Emulation: "A Second Iteration of NaivPhys4RP for Learningless and Safe Recognition and 6D-Pose Estimation of (Transparent) Objects"

• Franklin Kenghagho Kenfack
• Michael Neumann
• Patrick Mania
• Michael Beetz

In our previous work, we designed a human-like white-box and causal generative model of perception NaivPhys4RP, essentially based on cognitive emulation to understand the past, the present and the future of the state of complex worlds from poor observations. In this paper, as recommended in that previous work, we first refine the theoretical model of NaivPhys4RP in terms of integration of variables as well as perceptual inference tasks to solve. Intuitively, the system is closed under the injection, update and dependency of variables. Then, we present a first implementation of NaivPhys4RP that demonstrates the learningless and safe recognition and 6D-Pose estimation of objects from poor sensor data (e. g. , occlusion, transparency, poor-depth, in-hand). This does not only make a substantial step forward comparatively to classical perception systems in perceiving objects in these scenarios, but escape the burden of data-intensive learning and operate safely (transparency and causality — we fit sensor data into mentally constructed meaningful worlds). With respect to ChatGPT’s ambitions, it can imagine physico-realistic socio-physical scenes from texts, demonstrate understanding of these texts, and all these with no data- and resource-intensive learning.

ICRA Conference 2024 Conference Paper

RoboGrind: Intuitive and Interactive Surface Treatment with Industrial Robots

• Benjamin Alt
• Florian Stöckl
• Silvan Müller
• Christopher Braun
• Julian Raible
• Saad Alhasan
• Oliver Rettig
• Lukas Ringle

Surface treatment tasks such as grinding, sanding or polishing are a vital step of the value chain in many industries, but are notoriously challenging to automate. We present RoboGrind, an integrated system for the intuitive, interactive automation of surface treatment tasks with industrial robots. It combines a sophisticated 3D perception pipeline for surface scanning and automatic defect identification, an interactive voice-controlled wizard system for the AI-assisted bootstrapping and parameterization of robot programs, and an automatic planning and execution pipeline for force-controlled robotic surface treatment. RoboGrind is evaluated both under laboratory and real-world conditions in the context of refabricating fiberglass wind turbine blades.

AAMAS Conference 2024 Conference Paper

Simulated Robotic Soft Body Manipulation

• Glareh Mir
• Michael Beetz

The performance of intelligent agents manipulating a soft body object depends on the agent’s understanding of the execution environment. Hence, by keeping the agent fixed and changing the environment, the difference between environments can be measured. However, this becomes complicated when dealing with agents that learn in each environment. We propose a framework for evaluating the influence of the simulated soft bodies (and related object models) on the reinforcement learning algorithms’ performance. The change in algorithm behavior is quantified between different environments, and the correlation of behavioral difference is measured via statistical analysis. An evaluation case is presented on PyBullet and MuJoCo physics simulation environments with DDPG, PPO, TD3 and SAC algorithms.

ICRA Conference 2024 Conference Paper

Translating Universal Scene Descriptions into Knowledge Graphs for Robotic Environment

• Giang Hoang Nguyen
• Daniel Beßler
• Simon Stelter
• Mihai Pomarlan
• Michael Beetz

Robots performing human-scale manipulation tasks require an extensive amount of knowledge about their surroundings in order to perform their actions competently and human-like. In this work, we investigate the use of virtual reality technology as an implementation for robot environment modeling, and present a technique for translating scene graphs into knowledge bases. To this end, we take advantage of the Universal Scene Description (USD) format which is an emerging standard for the authoring, visualization and simulation of complex environments. We investigate the conversion of USD-based environment models into Knowledge Graph (KG) representations that facilitate semantic querying and integration with additional knowledge sources. The contributions of the paper are validated through an application scenario in the service robotics domain.

KR Conference 2023 Conference Paper

Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations

• Benjamin Alt
• Franklin Kenghagho Kenfack
• Andrei Haidu
• Darko Katic
• Rainer Jäkel
• Michael Beetz

Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https: //github. com/ease-crc/vr-program-synthesis.

AAMAS Conference 2023 Conference Paper

Shopping Assistance for Everyone: Dynamic Query Generation On a Semantic Digital Twin As a Basis for Autonomous Shopping Assistance

• Michaela Kümpel
• Jonas Dech
• Alina Hawkin
• Michael Beetz

While the Digital Twin technology can be used by robotic agents to autonomously digitise retail stores, the Semantic Web offers vast machine-understandable product information that can be utilised by both digital and robotic agents. We propose connecting shopping assistants to a semantic Digital Twin for a service-oriented shopping experience. The semantic Digital Twin connects product information from the Semantic Web to retail environment information created by an autonomous robot performing stocktaking. It can be used to retrieve relevant information for action execution by shopping assistants that dynamically generate queries to answer complex questions like “Where is toothpaste from (my preferred brand) containing natural ingredients? ", thus making the contained knowledge actionable.

ECAI Conference 2023 Conference Paper

Towards a Neuronally Consistent Ontology for Robotic Agents

• Florian Ahrens
• Mihai Pomarlan
• Daniel Beßler
• Thorsten Fehr
• Michael Beetz
• Manfred Herrmann

The Collaborative Research Center for Everyday Activity Science & Engineering (CRC EASE) aims to enable robots to perform environmental interaction tasks with close to human capacity. It therefore employs a shared ontology to model the activity of both kinds of agents, empowering robots to learn from human experiences. To properly describe these human experiences, the ontology will strongly benefit from incorporating characteristics of neuronal information processing which are not accessible from a behavioral perspective alone. We, therefore, propose the analysis of human neuroimaging data for evaluation and validation of concepts and events defined in the ontology model underlying most of the CRC projects. In an exploratory analysis, we employed an Independent Component Analysis (ICA) on functional Magnetic Resonance Imaging (fMRI) data from participants who were presented with the same complex video stimuli of activities as robotic and human agents in different environments and contexts. We then correlated the activity patterns of brain networks represented by derived components with timings of annotated event categories as defined by the ontology model. The present results demonstrate a subset of common networks with stable correlations and specificity towards particular event classes and groups, associated with environmental and contextual factors. These neuronal characteristics will open up avenues for adapting the ontology model to be more consistent with human information processing.

IROS Conference 2022 Conference Paper

An open-source motion planning framework for mobile manipulators using constraint-based task space control with linear MPC

• Simon Stelter
• Georg Bartels
• Michael Beetz

We present an open source motion planning framework for ROS, which uses constraint and optimization based task space control to generate trajectories for the whole body of mobile manipulators. Motion goals are defined as constraints which are enforced on task space functions. They map the controllable degrees of freedom of a system onto custom task spaces, which can, but do not have to be, the Cartesian space. We use this expressive tool from motion control to pre-compute trajectories in order to utilize the fact that most robots offer controllers to follow such trajectories. As a result, our framework only requires a kinematic model of the robot to control it. In addition, we extend the constraint-based motion control approach with linear MPC to explicitly optimize for velocity, acceleration and jerk simultaneously, which allows us to enforce constraints on all derivatives in both joint and task space at the same time. As a result, we can reuse predefined motion goals on any robot without modifications. Our framework was tested on four different robots to show its generality.

AAMAS Conference 2022 Conference Paper

Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities

• Alexander Wich
• Holger Schultheis
• Michael Beetz

A key challenge for robotic systems is to figure out the behavior of another agent. The capability to draw correct inferences is crucial to derive human behavior from examples. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally (observational evidence). For this reason, robots that rely on inferences that are correlational risk a biased interpretation of the evidence. We propose equipping robots with the necessary tools to conduct observational studies on people. Specifically, we propose and explore the feasibility of structural causal models with non-parametric estimators to derive empirical estimates on hand behavior in the context of object manipulation in a virtual kitchen scenario. In particular, we focus on inferences under (the weaker) conditions of partial confounding (the model covering only some factors) and confront estimators with hundreds of samples instead of the typical order of thousands. Studying these conditions explores the boundaries of the approach and its viability. Despite the challenging conditions, the estimates inferred from the validation data are correct. Moreover, these estimates are stable against three refutation strategies where four estimators are in agreement. Furthermore, the causal quantity for two individuals reveals the sensibility of the approach to detect positive and negative effects. The validity, stability, and explainability of the approach are encouraging and serve as the foundation for further research.

IROS Conference 2022 Conference Paper

Heuristic-free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments

• Benjamin Alt
• Darko Katic
• Rainer Jäkel
• Michael Beetz

In both industrial and service domains, a central benefit of the use of robots is their ability to quickly and reliably execute repetitive tasks. However, even relatively simple peg-in-hole tasks are typically subject to stochastic variations, requiring search motions to find relevant features such as holes. While search improves robustness, it comes at the cost of increased runtime: More exhaustive search will maximize the probability of successfully executing a given task, but will significantly delay any downstream tasks. This trade-off is typically resolved by human experts according to simple heuristics, which are rarely optimal. This paper introduces an automatic, data-driven and heuristic-free approach to optimize robot search strategies. By training a neural model of the search strategy on a large set of simulated stochastic environments, conditioning it on few real-world examples and inverting the model, we can infer search strategies which adapt to the time-variant characteristics of the underlying probability distributions, while requiring very few real-world measurements. We evaluate our approach on two different industrial robots in the context of spiral and probe search for THT electronics assembly. **See github.com/benjaminalt/dpse for code and data.

ICRA Conference 2021 Conference Paper

Automated acquisition of structured, semantic models of manipulation activities from human VR demonstration

• Andrei Haidu
• Michael Beetz

In this paper we present a system capable of collecting and annotating, human performed, robot understandable, everyday activities from virtual environments. The human movements are mapped in the simulated world using off-the-shelf virtual reality devices with full body, and eye tracking capabilities. All the interactions in the virtual world are physically simulated, thus movements and their effects are closely relatable to the real world. During the activity execution, a subsymbolic data logger is recording the environment and the human gaze on a per-frame basis, enabling offline scene reproduction and replays. Coupled with the physics engine, online monitors (symbolic data loggers) are parsing (using various grammars) and recording events, actions, and their effects in the simulated world.

IROS Conference 2021 Conference Paper

Imagination-enabled Robot Perception

• Patrick Mania
• Franklin Kenghagho Kenfack
• Michael Neumann
• Michael Beetz

Many of today’s robot perception systems aim at accomplishing perception tasks that are too simplistic and too hard. They are too simplistic because they do not require the perception systems to provide all the information needed to accomplish manipulation tasks. Typically the perception results do not include information about the part structure of objects, articulation mechanisms and other attributes needed for adapting manipulation behavior. On the other hand, the perception problems stated are also too hard because — unlike humans— the perception systems cannot leverage the expectations about what they will see to their full potential. Therefore, we investigate a variation of robot perception tasks suitable for robots accomplishing everyday manipulation tasks, such as household robots or a robot in a retail store. In such settings it is reasonable to assume that robots know most objects and have detailed models of them. We propose a perception system that maintains its beliefs about its environment as a scene graph with physics simulation and visual rendering. When detecting objects, the perception system retrieves the model of the object and places it at the corresponding place in a VR-based environment model. The physics simulation ensures that object detections that are physically not possible are rejected and scenes can be rendered to generate expectations at the image level. The result is a perception system that can provide useful information for manipulation tasks.

ICRA Conference 2021 Conference Paper

Robot Program Parameter Inference via Differentiable Shadow Program Inversion

• Benjamin Alt
• Darko Katic
• Rainer Jäkel
• Asil Kaan Bozcuoglu
• Michael Beetz

Challenging manipulation tasks can be solved effectively by combining individual robot skills, which must be parameterized for the concrete physical environment and task at hand. This is time-consuming and difficult for human programmers, particularly for force-controlled skills. To this end, we present Shadow Program Inversion (SPI), a novel approach to infer optimal skill parameters directly from data. SPI leverages unsupervised learning to train an auxiliary differentiable program representation ("shadow program") and realizes parameter inference via gradient-based model inversion. Our method enables the use of efficient first-order optimizers to infer optimal parameters for originally non-differentiable skills, including many skill variants currently used in production. SPI zero-shot generalizes across task objectives, meaning that shadow programs do not need to be retrained to infer parameters for different task variants. We evaluate our methods on three different robots and skill frameworks in industrial and household scenarios. Code and examples are available at https://innolab.artiminds.com/icra2021.

ICRA Conference 2021 Conference Paper

The Robot Household Marathon Experiment

• Gayane Kazhoyan
• Simon Stelter
• Franklin Kenghagho Kenfack
• Sebastian Koralewski
• Michael Beetz

In this paper, we present an experiment, designed to investigate and evaluate the scalability and the robustness aspects of mobile manipulation. The experiment involves performing variations of mobile pick and place actions and opening/closing environment containers in a human household. The robot is expected to act completely autonomously for extended periods of time. We discuss the scientific challenges raised by the experiment as well as present our robotic system that can address these challenges and successfully perform all the tasks of the experiment. We present empirical results and the lessons learned as well as discuss where we hit limitations.

ECAI Conference 2020 Conference Paper

A Formal Model of Affordances for Flexible Robotic Task Execution

• Daniel Beßler
• Robert Porzel
• Mihai Pomarlan
• Michael Beetz
• Rainer Malaka
• John A. Bateman

One of the key reasoning tasks of robotic agents is inferring possible actions that can be accomplished with a given object at hand. This cognitive task is commonly referred to as inferring the affordances of objects. In this paper, we propose a novel conceptualization of affordances and its realization as a description logic ontology. The key idea of the framework is that it proposes candidate affordances through inference, and that these can then be validated through physics-based simulation. We showcase the practical use of our conceptualization by means of demonstrating what competency questions an agent equipped with it can answer. The proposed formal model is implemented as a TBox OWL ontology of affordances based on the DOLCE Ultra Light + DnS foundational ontology.

KER Journal 2020 Journal Article

A review and comparison of ontology-based approaches to robot autonomy – ADDENDUM

• Alberto Olivares-Alarcos
• Daniel Beßler
• Alaa Khamis
• Paulo Goncalves
• Maki K. Habib
• Julita Bermejo-Alonso
• Marcos Barreto
• Mohammed Diab

IROS Conference 2020 Conference Paper

Learning Motion Parameterizations of Mobile Pick and Place Actions from Observing Humans in Virtual Environments

• Gayane Kazhoyan
• Alina Hawkin
• Sebastian Koralewski
• Andrei Haidu
• Michael Beetz

In this paper, we present an approach and an implemented pipeline for transferring data acquired from observing humans in virtual environments onto robots acting in the real world, and adapting the data accordingly to achieve successful task execution. We demonstrate our pipeline by inferring seven different symbolic and subsymbolic motion parameters of mobile pick and place actions, which allows the robot to set a simple breakfast table. We propose an approach to learn general motion parameter models and discuss, which parameters can be learned at which abstraction level.

IROS Conference 2020 Conference Paper

RobotVQA - A Scene-Graph- and Deep-Learning-based Visual Question Answering System for Robot Manipulation

• Franklin Kenghagho Kenfack
• Feroz Ahmed Siddiky
• Ferenc Balint-Benczedi
• Michael Beetz

Visual robot perception has been challenging to successful robot manipulation in noisy, cluttered and dynamic environments. While some perception systems fail to provide an adequate semantics of the scene, others fail to present appropriate learning models and training data. Another major issue encountered in some robot perception systems is their inability to promptly respond to robot control programs whose realtimeness is crucial. This paper proposes an architecture to robot vision for manipulation tasks that addresses the three issues mentioned above. The architecture encompasses a generator of training datasets and a learnable scene describer, coined as RobotVQA for Robot Visual Question Answering. The architecture leverages the power of deep learning to predict and photo-realistic virtual worlds to train. RobotVQA takes as input a robot scene's RGB or RGBD image, detects all relevant objects in it, then describes in realtime each object in terms of category, color, material, shape, openability, 6D-pose and segmentation mask. Moreover, RobotVQA computes the qualitative spatial relations among those objects. We refer to such a scene description in this paper as scene graph or semantic graph of the scene. In RobotVQA, prediction and training take place in a unified manner. Finally, we demonstrate how RobotVQA is suitable for robot control systems that interpret perception as a question answering process.

ICRA Conference 2020 Conference Paper

Towards Plan Transformations for Real-World Mobile Fetch and Place

• Gayane Kazhoyan
• Arthur Niedzwiecki
• Michael Beetz

In this paper, we present an approach and an implemented framework for applying plan transformations to real-world mobile manipulation plans, in order to specialize them to the specific situation at hand. The framework can improve execution cost and achieve better performance by autonomously transforming robot's behavior at runtime. To demonstrate the feasibility of our approach, we apply three example transformations to the plan of a PR2 robot performing simple table setting and cleaning tasks in the real world. Based on a large amount of experiments in a fast plan projection simulator, we make conclusions on improved execution performance.

ICRA Conference 2019 Conference Paper

A Framework for Self-Training Perceptual Agents in Simulated Photorealistic Environments

• Patrick Mania
• Michael Beetz

The development of high-performance perception for mobile robotic agents is still challenging. Learning appropriate perception models usually requires extensive amounts of labeled training data that ideally follows the same distribution as the data an agent will encounter in its target task. Recent developments in gaming industry led to game engines able to generate photorealistic environments in real-time, which can be used to realistically simulate the sensory input of an agent. We propose a novel framework which allows the definition of different learning scenarios and instantiates these scenarios in a high quality game engine where a perceptual agent can act and learn in. The scenarios are specified in a newly developed scenario description language that allows the parametrization of the virtual environment and the perceptual agent. New scenarios can be sampled from a task-specific object distribution that allows the automatic generation of extensive amounts of different learning environments for the perceptual agent. We will demonstrate the plausibility of the framework by conducting object recognition experiments on a real robotic system which has been trained within our framework.

KER Journal 2019 Journal Article

A review and comparison of ontology-based approaches to robot autonomy

• Alberto Olivares-Alarcos
• Daniel Beßler
• Alaa Khamis
• Paulo Goncalves
• Maki K. Habib
• Julita Bermejo-Alonso
• Marcos Barreto
• Mohammed Diab

Abstract Within the next decades, robots will need to be able to execute a large variety of tasks autonomously in a large variety of environments. To relax the resulting programming effort, a knowledge-enabled approach to robot programming can be adopted to organize information in re-usable knowledge pieces. However, for the ease of reuse, there needs to be an agreement on the meaning of terms. A common approach is to represent these terms using ontology languages that conceptualize the respective domain. In this work, we will review projects that use ontologies to support robot autonomy. We will systematically search for projects that fulfill a set of inclusion criteria and compare them with each other with respect to the scope of their ontology, what types of cognitive capabilities are supported by the use of ontologies, and which is their application domain.

ICRA Conference 2019 Conference Paper

Adapting Everyday Manipulation Skills to Varied Scenarios

• Pawel Gajewski
• Paulo Abelha
• Georg Bartels
• Chaozheng Wang
• Frank Guerin
• Bipin Indurkhya
• Michael Beetz
• Bartlomiej Sniezynski

We address the problem of executing tool-using manipulation skills in scenarios where the objects to be used may vary. We assume that point clouds of the tool and target object can be obtained, but no interpretation or further knowledge about these objects is provided. The system must interpret the point clouds and decide how to use the tool to complete a manipulation task with a target object; this means it must adjust motion trajectories appropriately to complete the task. We tackle three everyday manipulations: scraping material from a tool into a container, cutting, and scooping from a container. Our solution encodes these manipulation skills in a generic way, with parameters that can be filled in at run-time via queries to a robot perception module; the perception module abstracts the functional parts of the tool and extracts key parameters that are needed for the task. The approach is evaluated in simulation and with selected examples on a PR2 robot.

ICRA Conference 2019 Conference Paper

Automated Models of Human Everyday Activity based on Game and Virtual Reality Technology

• Andrei Haidu
• Michael Beetz

In this paper, we will describe AMEvA (Automated Models of Everyday Activities), a special-purpose knowledge acquisition, interpretation, and processing system for human everyday manipulation activity that can automatically: (1) create and simulate virtual human living and working environments (such as kitchens and apartments) with a scope, extent, level of detail, physics, and close to photorealism that facilitates and promotes the natural and realistic execution of human everyday manipulation activities; (2) record human manipulation activities performed in the respective virtual reality environment as well as their effects on the environment and detect force-dynamic states and events; (3) decompose and segment the recorded activity data into meaningful motions and categorize the motions according to action models used in cognitive science; and (4) represent the interpreted activities symbolically in KNOWROB [1] using a first-order time interval logic representation.

IROS Conference 2019 Conference Paper

Continuous Modeling of Affordances in a Symbolic Knowledge Base

• Asil Kaan Bozcuoglu
• Yuki Furuta
• Kei Okada
• Michael Beetz
• Masayuki Inaba

As robots start to execute complex manipulation tasks, they are expected to improve their skill set over time as humans do. A prominent approach to accomplish this is having robots to keep models of their actions based on their experiences in order to improve their action executions in the future. In this paper, we present such a methodology where robots start to execute some actions with random parameters and record their generic execution logs with semantic annotations in a symbolic knowledge base for robots. Using the data inside logs, multivariate Gaussian mixture models are fitted to the high-level action parameters for later executions. These affordance models are being updated whenever a new execution is carried on. In essence, robots can use these continuously-updated probabilistic model for improving their actions To prove the applicability we demonstrate opening-a-fridge-door experiments with a PR2 robot.

IROS Conference 2019 Conference Paper

Executing Underspecified Actions in Real World Based on Online Projection

• Gayane Kazhoyan
• Michael Beetz

Plan execution on real robots in realistic environments is underdetermined and often leads to failures. The choice of action parameterization is crucial for task success. In this paper, we present a mechanism for a robot that is acting in a real-world environment to think ahead of time with fast plan projection and, thereby, choose action parameterizations that are predicted to lead to successful execution. For finding causal relationships between action parameterizations and task success, we provide the robot with means for plan introspection and propose a systematic and hierarchical plan structure to support that. We evaluate our approach by showing how a PR2 robot, when equipped with the proposed system, is able to choose action parameterizations that increase task execution success rates and overall performance of fetch and place actions in a real world setting.

IROS Conference 2018 Conference Paper

Cognition-enabled Framework for Mixed Human-Robot Rescue Teams

• Fereshta Yazdani
• Gayane Kazhoyan
• Asil Kaan Bozcuoglu
• Andrei Haidu
• Ferenc Balint-Benczedi
• Daniel Beßler
• Mihai Pomarlan
• Michael Beetz

With the advancements in robotic technology and the progress in human-robot interaction research, the interest in deploying mixed human-robot teams in rescue missions is increasing. Due to their complementary capabilities in terms of locomotion, visibility and reachability of areas, human-robot teams are considerably deployed in real-world settings, albeit the robotic agents in such scenarios are normally fully teleoperated. A major barrier to successful and efficient mission execution in those teams is the lack of cognitive skills in robotic systems. In this paper, we present a cognition-enabled framework and an implemented system where robotic agents are equipped with cognitive capabilities to naturally communicate with humans and autonomously perform tasks. The framework allows for natural tasking of robots, reasoning about robot behavior, capabilities and actions, and a common belief state representation for shared mission awareness of robots and human operators.

ICRA Conference 2018 Conference Paper

Configuration of Perception Systems via Planning Over Factor Graphs

• Vincent Dietrich
• Bernd Kast
• Philipp S. Schmitt
• Sebastian Albrecht 0001
• Michael Fiegert
• Wendelin Feiten
• Michael Beetz

Sensor guided, automated systems require the composition of various sensors and data processing algorithms to obtain relevant information for performing their task. Many applications have additional requirements such as a certain accuracy, which has to be achieved despite sensor noise and calibration errors. In this paper we model the configuration of perception systems as a planning problem over probabilistic graphical models. We work on a subset of the full configuration space of perceptions systems, specifically the used sensors, data processing algorithms and view poses. Based on a semantic description of the goal, available sensors and data processing algorithms, our system plans perception steps and sensor data fusion autonomously. The planner operates by constructing a factor graph until the accuracy requirements of tasks are fulfilled or unobtainable with the available action set. We validate our approach in an industrial assembly scenario.

ICRA Conference 2018 Conference Paper

Know Rob 2. 0 - A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents

• Michael Beetz
• Daniel Beßler
• Andrei Haidu
• Mihai Pomarlan
• Asil Kaan Bozcuoglu
• Georg Bartels

In this paper we present KnowRob2, a second generation knowledge representation and reasoning framework for robotic agents. KnowRob2 is an extension and partial redesign of KnowRob, currently one of the most advanced knowledge processing systems for robots that has enabled them to successfully perform complex manipulation tasks such as making pizza, conducting chemical experiments, and setting tables. The knowledge base appears to be a conventional first-order time interval logic knowledge base, but it exists to a large part only virtually: many logical expressions are constructed on demand from data structures of the control program, computed through robotics algorithms including ones for motion planning and solving inverse kinematics problems, and log data stored in noSQL databases. Novel features and extensions of KnowRob2 substantially increase the capabilities of robotic agents of acquiring open-ended manipulation skills and competence, reasoning about how to perform manipulation actions more realistically, and acquiring commonsense knowledge.

IROS Conference 2018 Conference Paper

KnowRobSIM - Game Engine-Enabled Knowledge Processing Towards Cognition-Enabled Robot Control

• Andrei Haidu
• Daniel Beßler
• Asil Kaan Bozcuoglu
• Michael Beetz

AI knowledge representation and reasoning methods consider actions to be blackboxes that abstract away from how they are executed. This abstract view does not suffice for the decision making capabilities required by robotic agents that are to accomplish manipulation tasks. Such robots have to reason about how to pour without spilling, where to grasp a pot, how to open different containers, and so on. To enable such reasoning it is necessary to consider how objects are perceived, how motions can be executed and parameterized, and how motion parameterization affects the physical effects of actions. To this end, we propose to complement and extend symbolic reasoning methods with KnowRob SIM, an additional reasoning infrastructure based on modern game engine technology, including the subsymbolic world modeling through data structures, action simulation based on physics engine, and world scene rendering. We demonstrate how KnowRob SIM can perform powerful reasoning, prediction, and learning tasks that are required for informed decision making in object manipulation.

AAMAS Conference 2018 Conference Paper

OWL-enabled Assembly Planning for Robotic Agents

• Daniel Be�ler
• Mihai Pomarlan
• Michael Beetz

Assembly cells run by intelligent robotic agents promise highly flexible product customization without the cost implication product individualization has nowadays. One of the main questions an assembly robot has to answer is which sequence of manipulation actions it should perform to create an assembled product from scattered pieces available. We propose a novel approach to assembly planning that employs Description Logics (DL) to describe what an assembled product should look like, and to plan the next action according to faulty and missing assertions in the robot’s beliefs about an ongoing assembly task. To this end we extend the KNOWROB knowledge base with representations and inference rules that enable robots to reason about incomplete assemblies. We show that our approach performs well for large batches of assembly pieces available, as well as for varying structural complexity of assembled products.

IROS Conference 2018 Conference Paper

Reasoning Systems for Semantic Navigation in Mobile Robots

• Jonathan Crespo
• Ramón Barber
• Óscar Martínez Mozos
• Daniel Beßler
• Michael Beetz

Semantic navigation is the navigation paradigm in which environmental semantic concepts and their relationships are taken into account to plan the route of a mobile robot. This paradigm facilitates the interaction with humans and the understanding of human environments in terms of navigation goals and tasks. At the high level, a semantic navigation system requires two main components: a semantic representation of the environment, and a reasoning system. This paper is focused on develop a model of the environment using semantic concepts. This paper presents two solutions for the semantic navigation paradigm. Both systems implement an ontological model. Whilst the first one uses a relational database, the second one is based on KnowRob. Both systems have been integrated in a semantic navigator. We compare both systems at the qualitative and quantitative levels, and present an implementation on a mobile robot as a proof of concept.

ICRA Conference 2018 Conference Paper

The Exchange of Knowledge Using Cloud Robotics

• Asil Kaan Bozcuoglu
• Gayane Kazhoyan
• Yuki Furuta
• Simon Stelter
• Michael Beetz
• Kei Okada
• Masayuki Inaba

To enable robots to perform human-level tasks flexibly in varying conditions, we need a mechanism that allows them to exchange knowledge between themselves for crowd-sourcing the knowledge gap problem. One approach to achieve this is to equip a cloud application with a range of encyclopedic knowledge (i. e. ontologies) and execution logs of different robots performing the same tasks in different environments. In this paper, we show how knowledge exchange between robots can be done using OPENEASE as the cloud application. We equipped OPENEASE with ontologies about the kitchen domain, execution logs of three robots operating in two different kitchens, and semantic descriptions of both environments. By addressing two different use cases, we show that two PR2 robots and one Fetch robot can successfully adapt each other's plan parameters and sub symbolic data to the experiments that they are conducting.

IROS Conference 2018 Conference Paper

Variations on a Theme: "It's a Poor Sort of Memory that Only Works Backwards"

• Ferenc Balint-Benczedi
• Michael Beetz

Adapting the perceptual capabilities of mobile robots to new objects or new environments can be a time consuming task. In this paper we focus on specializing perceptual capabilities of mobile robots to new objects through a knowledge based, virtual scene rendering approach. Episodic memories of a robotic agent, gathered during the execution of a task are considered to be the main "theme". Variations of this theme are then generated based on background knowledge about the objects and data gathered with the purpose of learning new models for detection and recognition. We demonstrate the applicability of our approach by adapting the perceptual capabilities of a mobile robot performing pick and place tasks, to recognize new sets of objects.

ICRA Conference 2017 Conference Paper

A cloud service for robotic mental simulations

• Asil Kaan Bozcuoglu
• Michael Beetz

Robotic agents that do everyday manipulation tasks can hugely benefit from being able to predict consequences of their actions just before the execution. However, such a simulation technique is usually computationally-expensive and may not be achieved with agents' self computing power. For this problem, cloud robotics may offer a solution. Cloud robotics is an emerging field in the intersection of robotics and cloud computing which enables robots to access a greater amount of processing power and storage capacity than it can employ within itself. In this work, we introduce a mental simulation service to, one of the cloud engines, openEASE [1]. Using this service, researchers and robots can describe the world model, the state of the agent and the problem that is being dealt with. In return, it simulates the world and runs a learning algorithm and suggests a solution how the robotic agent can handle the problem. This service does not only offer a free remote access to simulation which is computationally expensive but also thanks to OPEnEASE's rich reasoning techniques these simulated experiments can be reasoned later on using prolog queries.

AAMAS Conference 2017 Conference Paper

Cognition-enabled Task Interpretation for Human-Robot Teams in a Simulation-based Search and Rescue Mission

• Fereshta Yazdani
• Matthias Scheutz
• Michael Beetz

AAMAS Conference 2017 Conference Paper

Deeper Understanding of Vague Instructions through Simulated Execution

• Mihai Pomarlan
• Daniel Nyga
• Mareike Picklum
• Sebastian Koralewski
• Michael Beetz

A commonsense understanding of the physical world will be crucial for the robots of the future as they strive to perform everyday activities and instructions formulated by human users in natural language. One mechanism that is believed to assist human cognition in commonsense reasoning is mental simulation, the envisioning of actions before they are performed. We therefore present a system integrating simulation of robot plans with probabilistic reasoning about natural-language instructions, to create a complete pipeline from instruction to execution to storing and analyzing results of the simulation. This integration allows the robotic system to efficiently infer knowledge about the physical world that would be tedious to specify by hand in a collection of logical statements. Our system will be available online1 for open use by researchers.

AIJ Journal 2017 Journal Article

Envisioning the qualitative effects of robot manipulation actions using simulation-based projections

• Lars Kunze
• Michael Beetz

ICRA Conference 2017 Conference Paper

Instruction completion through instance-based learning and semantic analogical reasoning

• Daniel Nyga
• Mareike Picklum
• Sebastian Koralewski
• Michael Beetz

As autonomous, mobile robots are increasingly entering our everyday lives and the tasks they are to perform are getting continuously more complex and versatile, instructing robots by means of natural-language commands becomes more and more important. Such instructions, stated by humans and originally intended for human use, are typically formulated very vaguely and lack critical information about how to perform particular actions. Probabilistic relational models have shown promise in filling in missing information pieces that have been omitted in such instructions. However, the enormous size of these models and the computational expense in learning and reasoning often impedes their practical applicability to real-world domains. In this work, we propose a novel instance-based learning approach towards building up knowledge bases for instruction completion, which combines probabilistic methods with semantic analogical reasoning. Probabilistic reasoning is employed to build up a knowledge base of natural-language instruction sheets, while instruction completion can be achieved through fast database queries. We showcase the scalabilty of our approach by building up a KB of more than 100, 000 instruction steps that have been mined from the wikihow.com web site and which are publicly accessible from within the Prac [21] natural-language interpreter.

IROS Conference 2017 Conference Paper

Programming robotic agents with action descriptions

• Gayane Kazhoyan
• Michael Beetz

This paper tackles the problem of generalizing robot control programs over multiple objects, tasks and environments, based on the concept of action descriptions. These are abstract, general, semantic descriptions of an action that are augmented during execution with subsymbolic parameters specific to the context at hand. The parameters are inferred through reasoning rules, which extract the context from the action description and the belief state of the robot. The proposed system scales well with increasing number of reasoning rules required to support the knowledge-intensive manipulation tasks. The architecture combines the high-level robot control program with the reasoning engine in a modular way, thus improving the scalability of the system. The approach is validated in the context of setting a table with a PR2 robot.

AIJ Journal 2017 Journal Article

Representations for robot knowledge in the KnowRob framework

• Moritz Tenorth
• Michael Beetz

AAMAS Conference 2017 Conference Paper

Task Parametrization through Multi-modal Analysis of Robot Experiences

• Jan Winkler
• Asil Kaan Bozcuoglu
• Mihai Pomarlan
• Michael Beetz

With quickly progressing and increasingly complex robot control and reasoning systems, a large gap of practical real-world knowledge for robots needs to be filled. While two prominent directions exist, namely designing all knowledge manually, or completely bootstrapping it, we emphasize the combination of both: Starting with simple heuristics, we let robots explore a task, record memories, interpret their findings, and improve their own multi-modal understanding to better their own performance. In this work, we present a software system for autonomous robots that allows them to learn task nuances, and make informed decisions based on experience. They store these comprehensive probabilistic models of any task they perform in a robot knowledge service, benefiting from a shared knowledge base and centralized, well-maintained reasoning algorithms.

AIJ Journal 2017 Journal Article

Transferring skills to humanoid robots by extracting semantic representations from observations of human activities

• Karinne Ramirez-Amaro
• Michael Beetz
• Gordon Cheng

ICRA Conference 2017 Conference Paper

What no robot has seen before - Probabilistic interpretation of natural-language object descriptions

• Daniel Nyga
• Mareike Picklum
• Michael Beetz

We investigate the task of recognizing objects of daily use in human environments purely based on object descriptions given in natural language. In particular, we present an approach to transform phrases stated in natural language that describe such objects by their visual appearance into formal, semantic representations of their perceptual characteristics, which in turn can be used in a robot perception system in order to identify objects that the robot has never encountered before. To this end, we learn probabilistic first-order knowledge bases from encyclopedic articles and online dictionaries, which contain textual descriptions of a vast amount of everyday objects. We demonstrate the applicability of the approach on a robotic system in a proof-of-concept evaluation on a selected set of object descriptions acquired from the internet.

IROS Conference 2016 Conference Paper

Action recognition and interpretation from virtual demonstrations

• Andrei Haidu
• Michael Beetz

To properly perform tasks based on abstract instructions, autonomous robots need refined reasoning skills in order to bridge the gap between the ambiguous descriptions and the comprehensive information needed to execute the implied actions. In this article, we present an automated knowledge acquisition system from human executed tasks in virtual environments, and extend the knowledge processing system KNOWROB[1] to be capable to reason on the acquired data. We have set up two scenarios in a physics based simulator: creating a pancake, and garnishing a pizza dough. Users where asked to execute these tasks using the provided tools and ingredients. Using a data processing module we then collect the low-level data and the relevant abstract events from the performed episodes. The recorded data is then made available in a format that robots can understand, by using a symbolic layer to interconnect the two data types in a seamless way.

AAMAS Conference 2016 Conference Paper

Knowledge-Enabled Robotic Agents for Shelf Replenishment in Cluttered Retail Environments (Extended Abstract)

• Jan Winkler
• Ferenc Balint-Benczedi
• Thiemo Wiedemeyer
• Michael Beetz
• Narunas Vaskevicius
• Christian A. Mueller
• Tobias Fromm
• Andreas Birk

Autonomous robots in unstructured and dynamically changing retail environments have to master complex perception, knowledge processing, and manipulation tasks. To enable them to act competently, we propose a framework based on three core components: (◦) a knowledge-enabled perception system, capable of combining diverse information sources to cope with occlusions and stacked objects with a variety of textures and shapes, (◦) knowledge processing methods produce strategies for tidying up supermarket racks, and (◦) the necessary manipulation skills in confined spaces to arrange objects in semi-accessible rack shelves. We demonstrate our framework in an simulated environment as well as on a real shopping rack using a PR2 robot. Typical supermarket products are detected and rearranged in the retail rack, tidying up what was found to be misplaced items. General Terms Algorithms, Experimentation

IROS Conference 2016 Conference Paper

Learning models for constraint-based motion parameterization from interactive physics-based simulation

• Zhou Fang
• Georg Bartels
• Michael Beetz

For robotic agents to perform manipulation tasks in human environments at a human level or higher, they need to be able to relate the physical effects of their actions to how they are executing them; small variations in execution can have very different consequences. This paper proposes a framework for acquiring and applying action knowledge from naive user demonstrations in an interactive simulation environment under varying conditions. The framework combines a flexible constraint-based motion control approach with games-with-a-purpose-based learning using Random Forest Regression. The acquired action models are able to produce context-sensitive constraint-based motion descriptions to perform the learned action. A pouring experiment is conducted to test the feasibility of the suggested approach and shows the learned system can perform comparable to its human demonstrators.

ICRA Conference 2016 Conference Paper

Open robotics research using web-based knowledge services

• Michael Beetz
• Daniel Beßler
• Jan Oliver Winkler
• Jan-Hendrik Worch
• Ferenc Balint-Benczedi
• Georg Bartels
• Aude Billard
• Asil Kaan Bozcuoglu

In this paper we discuss how the combination of modern technologies in “big data” storage and management, knowledge representation and processing, cloud-based computation, and web technology can help the robotics community to establish and strengthen an open research discipline. We describe how we made the demonstrator of a EU project review openly available to the research community. Specifically, we recorded episodic memories with rich semantic annotations during a pizza preparation experiment in autonomous robot manipulation. Afterwards, we released them as an open knowledge base using the cloud- and web-based robot knowledge service OPENEASE. We discuss several ways on how this open data can be used to validate our experimental reports and to tackle novel challenging research problems.

AAMAS Conference 2016 Conference Paper

Robotic Agents Representing, Reasoning, and Executing Wiping Tasks for Daily Household Chores

• Daniel Leidner
• Wissam Bejjani
• Alin Albu-Schaeffer
• Michael Beetz

Universal robotic agents are envisaged to perform a wide range of manipulation tasks in everyday environments. A common action observed in many household chores is wiping, such as the absorption of spilled water with a sponge, skimming breadcrumbs off the dining table, or collecting shards of a broken mug using a broom. To cope with this versatility, the agents have to represent the tasks on a high level of abstraction. In this work, we propose to represent the medium in wiping tasks (e. g. water, breadcrumbs, or shards) as generic particle distribution. This representation enables us to represent wiping tasks as the desired state change of the particles, which allows the agent to reason about the effects of wiping motions in a qualitative manner. Based on this, we develop three prototypical wiping actions for the generic tasks of absorbing, collecting and skimming. The Cartesian wiping motions are resolved to joint motions exploiting the free degree of freedom of the involved tool. Furthermore, the workspace of the robotic manipulators is used to reason about the reachability of wiping motions. We evaluate our methods in simulated scenarios, as well as in a real experiment with the robotic agent Rollin’ Justin.

AAMAS Conference 2016 Conference Paper

Robots Reasoning with Cuts and Connections: Creating and Removing Entities (Extended Abstract)

• Mihai Pomarlan
• Michael Beetz

Several tasks that autonomous service robots will be expected to do involve changing material objects by cutting or separating parts, then rearranging them to obtain assemblages satisfying certain properties. In this paper we describe a system to represent and reason about entities that disappear or are created by the robot through such actions. Entities are grounded in objects that the robot can perceive and manipulate, and reasoning provides specific parameters for the robot’s actions. For this paper, our system has knowledge only of geometric aspects related to cutting and rearrangement of objects. We test our system in simulation, but also discuss how it can be connected to a robot’s perception and control. General Terms Algorithms, Languages, Theory

ICRA Conference 2016 Conference Paper

Scaling perception towards autonomous object manipulation - in knowledge lies the power

• Ferenc Balint-Benczedi
• Patrick Mania
• Michael Beetz

Mobile robots operating in a human environment face the challenge of recognizing objects that possess a multitude of different visual characteristics, affordances, and are found in visually challenging scenes. Because of this, perceptual capabilities of such robots need to go beyond detection or categorization of objects, and be able to answer queries not only about where certain objects are located based on their class label, but also about functional properties of these. To achieve an optimal performance, robots need to be aware of their environment, the task that they are to execute, and their perceptual capabilities. Given this knowledge, robotic agents need adequate mechanisms that apply the right method at the right time, in the right situation. In this paper we present a self-adaptive robotic perception system, that acts as a planner for task aware robot manipulation and enables querying on a broad domain. This is done through extending our existing perception framework, ROBOSHERLOCK, with the capability to adapt its perception pipelines based on the query, using knowledge-based reasoning. We will demonstrate the success of the approach, by presenting challenging queries, where the benefits of integrating knowledge processing into perception systems is shown.

IROS Conference 2015 Conference Paper

Classifying compliant manipulation tasks for automated planning in robotics

• Daniel Leidner
• Christoph Borst 0001
• Alexander Dietrich
• Michael Beetz
• Alin Albu-Schäffer

Many household chores and industrial manufacturing tasks require a certain compliant behavior to make deliberate physical contact with the environment. This compliant behavior can be implemented by modern robotic manipulators. However, in order to plan the task execution, a robot requires generic process models of these tasks which can be adapted to different domains and varying environmental conditions. In this work we propose a classification of compliant manipulation tasks meeting these requirements, to derive related actions for automated planning. We also present a classification for the sub-category of wiping tasks, which are most common and of great importance in service robotics. We categorize actions from an object-centric perspective to make them independent of any specific robot kinematics. The aim of the proposed taxonomy is to guide robotic programmers to develop generic actions for any kind of robotic systems in arbitrary domains.

IROS Conference 2015 Conference Paper

Learning action failure models from interactive physics-based simulations

• Andrei Haidu
• Daniel Kohlsdorf
• Michael Beetz

Predicting the outcome of an action can help a robot detect failures in advance, and schedule action replanning before an error occurs. We propose using an interactive physics based simulator with the aim of collecting realistic data to be used for learning. We then show how we save and query for specific information from the data more effectively. The data from the simulation is used to learn a failure detection model which is utilized by a real robot performing the same actions. We show that learning from simulation data is realistic enough to be applied on a real robot. The learning algorithm is more simple in design and outperforms the more complex one from our previous work.

IROS Conference 2015 Conference Paper

Multi-robot 6D graph SLAM connecting decoupled local reference filters

• Martin J. Schuster
• Christoph Brand
• Heiko Hirschmüller
• Michael Suppa
• Michael Beetz

Teams of mobile robots can be deployed in search and rescue missions to explore previously unknown environments. Methods for joint localization and mapping constitute the basis for (semi-)autonomous cooperative action, in particular when navigating in GPS-denied areas. As communication losses may occur, a decentralized solution is required. With these challenges in mind, we designed a submap-based SLAM system that relies on inertial measurements and stereo-vision to create multi-robot dense 3D maps. For online pose and map estimation, we integrate the results of keyframe-based local reference filters through incremental graph SLAM. To the best of our knowledge, we are the first to combine these two methods to benefit from their particular advantages for 6D multi-robot localization and mapping: Local reference filters on each robot provide real-time, long-term stable state estimates that are required for stabilization, control and fast obstacle avoidance, whereas online graph optimization provides global multi-robot pose and map estimates needed for cooperative planning. We propose a novel graph topology for a decoupled integration of local filter estimates from multiple robots into a SLAM graph according to the filters' uncertainty estimates and independence assumptions and evaluated its benefits on two different robots in indoor, outdoor and mixed scenarios. Further, we performed two extended experiments in a multi-robot setup to evaluate the full SLAM system, including visual robot detections and submap matches as inter-robot loop closure constraints.

ICRA Conference 2015 Conference Paper

Open-EASE

• Michael Beetz
• Moritz Tenorth
• Jan Oliver Winkler

Making future autonomous robots capable of accomplishing human-scale manipulation tasks requires us to equip them with knowledge and reasoning mechanisms. We propose Open-EASE, a remote knowledge representation and processing service that aims at facilitating these capabilities. Open-EASE gives its users unprecedented access to the knowledge of leading-edge autonomous robotic agents. It also provides the representational infrastructure to make inhomogeneous experience data from robots and human manipulation episodes semantically accessible, and is complemented by a suite of software tools that enable researchers and robots to interpret, analyze, visualize, and learn from the experience data. Using Open-EASE users can retrieve the memorized experiences of manipulation episodes and ask queries regarding to what the robot saw, reasoned, and did as well as how the robot did it, why, and what effects it caused.

ICRA Conference 2015 Conference Paper

RoboSherlock: Unstructured information processing for robot perception

• Michael Beetz
• Ferenc Balint-Benczedi
• Nico Blodow
• Daniel Nyga
• Thiemo Wiedemeyer
• Zoltán-Csaba Márton

We present RoboSherlock, an open source software framework for implementing perception systems for robots performing human-scale everyday manipulation tasks. In RoboSherlock, perception and interpretation of realistic scenes is formulated as an unstructured information management (UIM) problem. The application of the UIM principle supports the implementation of perception systems that can answer task-relevant queries about objects in a scene, boost object recognition performance by combining the strengths of multiple perception algorithms, support knowledge-enabled reasoning about objects and enable automatic and knowledge-driven generation of processing pipelines. We demonstrate the potential of the proposed framework by three feasibility studies of systems for real-world scene perception that have been built on top of RoboSherlock.

IROS Conference 2015 Conference Paper

Robot action plans that form and maintain expectations

• Jan Oliver Winkler
• Michael Beetz

Robots performing general purpose plans must deal with a wide variety of contexts. Situations they encounter might differ only in subtle, but important details in context and parameterization that have a massive impact on an action's outcome. To avoid the effort of encoding all possible combinations of subtleties into plans, we present a prediction framework that gives robot agents an intuition of their actions' effects and for choosing parameter values that have proven to be useful before. We let a robot form these predictions and expectations from episodic memories collected during earlier plan executions, improving its own behavior with every new situation encountered. We evaluate and explain our approach using experiments performed on a PR2 robot performing complex mobile manipulation activities in a kitchen environment.

IROS Conference 2015 Conference Paper

Robotic agents capable of natural and safe physical interaction with human co-workers

• Michael Beetz
• Georg Bartels
• Alin Albu-Schäffer
• Ferenc Balint-Benczedi
• Rico Belder
• Daniel Beßler
• Sami Haddadin
• Alexis Maldonado

Many future application scenarios of robotics envision robotic agents to be in close physical interaction with humans: On the factory floor, robotic agents shall support their human co-workers with the dull and health threatening parts of their jobs. In their homes, robotic agents shall enable people to stay independent, even if they have disabilities that require physical help in their daily life - a pressing need for our aging societies. A key requirement for such robotic agents is that they are safety-aware, that is, that they know when actions may hurt or threaten humans and actively refrain from performing them. Safe robot control systems are a current research focus in control theory. The control system designs, however, are a bit paranoid: programmers build “software fences” around people, effectively preventing physical interactions. To physically interact in a competent manner robotic agents have to reason about the task context, the human, and her intentions. In this paper, we propose to extend cognition-enabled robot control by introducing humans, physical interaction events, and safe movements as first class objects into the plan language. We show the power of the safety-aware control approach in a real-world scenario with a leading-edge autonomous manipulation platform. Finally, we share our experimental recordings through an online knowledge processing system, and invite the reader to explore the data with queries based on the concepts discussed in this paper.

IROS Conference 2015 Conference Paper

Towards robots conducting chemical experiments

• Gheorghe Lisca
• Daniel Nyga
• Ferenc Balint-Benczedi
• Hagen Langer
• Michael Beetz

Autonomous mobile robots are employed to perform increasingly complex tasks which require appropriate task descriptions, accurate object recognition, and dexterous object manipulation. In this paper we will address three key questions: How to obtain appropriate task descriptions from natural language (NL) instructions, how to choose the control program to perform a task description, and how to recognize and manipulate the objects referred by a task description? We describe an evaluated robotic agent which takes a natural language instruction stating a step of DNA extraction procedure as a starting point. The system is able to transform the textual instruction into an abstract symbolic plan representation. It can reason about the representation and answer queries about what, how, and why it is done. The robot selects the most appropriate control programs and robustly coordinates all manipulations required by the task description. The execution is based on a perception sub-system which is able to locate and recognize the objects and instruments needed in the DNA extraction procedure.

IROS Conference 2014 Conference Paper

Automatic segmentation and recognition of human activities from observation based on semantic reasoning

• Karinne Ramírez-Amaro
• Michael Beetz
• Gordon Cheng

Automatically segmenting and recognizing human activities from observations typically requires a very complex and sophisticated perception algorithm. Such systems would be unlikely implemented on-line into a physical system, such as a robot, due to the pre-processing step(s) that those vision systems usually demand. In this work, we present and demonstrate that with an appropriate semantic representation of the activity, and without such complex perception systems, it is sufficient to infer human activities from videos. First, we will present a method to extract the semantic rules based on three simple hand motions, i. e. move, not move and tool use. Additionally, the information of the object properties either ObjectActedOn or ObjectInHand are used. Such properties encapsulate the information of the current context. The above data is used to train a decision tree to obtain the semantic rules employed by a reasoning engine. This means, we extract lower-level information from videos and we reason about the intended human behaviors (high-level). The advantage of the abstract representation is that it allows to obtain more generic models out of human behaviors, even when the information is obtained from different scenarios. The results show that our system correctly segments and recognizes human behaviors with an accuracy of 85%. Another important aspect of our system is its scalability and adaptability toward new activities, which can be learned on-demand. Our system has been fully implemented on a humanoid robot, the iCub to experimentally validate the performance and the robustness of our system during on-line execution of the robot.

ICRA Conference 2014 Conference Paper

Controlled Natural Languages for language generation in artificial cognition

• Nicholas H. Kirk
• Daniel Nyga
• Michael Beetz

In this paper we discuss, within the context of artificial assistants performing everyday activities, a resolution method to disambiguate missing or not satisfactorily inferred action-specific information via explicit clarification. While arguing the lack of preexisting robot to human linguistic interaction methods, we introduce a novel use of Controlled Natural Languages (CNL) as means of output language and sentence construction for doubt verbalization. We additionally provide implemented working scenarios, state future possibilities and problems related to verbalization of technical cognition when making use of Controlled Natural Languages.

ECAI Conference 2014 Conference Paper

Knowledge-based Specification of Robot Motions

• Moritz Tenorth
• Georg Bartels
• Michael Beetz

In many cases, the success of a manipulation action performed by a robot is determined by how it is executed and by how the robot moves during the action. Examples are tasks such as unscrewing a bolt, pouring liquids and flipping a pancake. This aspect is often abstracted away in AI planning and action languages that assume that an action is successful as long as all preconditions are fulfilled. In this paper we investigate how constraint-based motion representations used in robot control can be combined with a semantic knowledge base in order to let a robot reason about movements and to automatically generate executable motion descriptions that can be adapted to different robots, objects and tools.

IROS Conference 2014 Conference Paper

Learning task outcome prediction for robot control from interactive environments

• Andrei Haidu
• Daniel Kohlsdorf
• Michael Beetz

In order to manage complex tasks such as cooking, future robots need to be action-aware and posses common sense knowledge. For example flipping a pancake requires a robot to know that a spatula has to be under a pancake in order to succeed. We present a novel approach for the extraction and learning of action and common sense knowledge, and developed a game using a robot-simulator with realistic physics for data acquisition. The game environment is a virtual kitchen, in which a user has to create a pancake by pouring pancake-mix on an oven and flipping it using a spatula. The interaction is done by controlling a virtual robot hand with a 3D input sensor. We incorporate a realistic fluid simulation in order to gather appropriate data of the pouring action. Furthermore, we present a task outcome prediction algorithm for this specific system and show how to learn a failure model for the pouring and flipping action.

ICRA Conference 2014 Conference Paper

PR2 looking at things - Ensemble learning for unstructured information processing with Markov logic networks

• Daniel Nyga
• Ferenc Balint-Benczedi
• Michael Beetz

We investigate the perception and reasoning task of answering queries about realistic scenes with objects of daily use perceived by a robot. A key problem implied by the task is the variety of perceivable properties of objects, such as their shape, texture, color, size, text pieces and logos, that go beyond the capabilities of individual state-of-the-art perception methods. A promising alternative is to employ combinations of more specialized perception methods. In this paper we propose a novel combination method, which structures perception in a two-step process, and apply this method in our object perception system. In a first step, specialized methods annotate detected object hypotheses with symbolic information pieces. In the second step, the given query Q is answered by inferring the conditional probability P(Q | E), where E are the symbolic information pieces considered as evidence for the conditional probability. In this setting Q and E are part of a probabilistic model of scenes, objects and their annotations, which the perception method has beforehand learned a joint probability distribution of. Our proposed method has substantial advantages over alternative methods in terms of the generality of queries that can be answered, the generation of information that can actively guide perception, the ease of extension, the possibility of including additional kinds of evidences, and its potential for the realization of self-improving and — specializing perception systems. We show for object categorization, which is a subclass of the probabilistic inferences, that impressive categorization performance can be achieved combining the employed expert perception methods in a synergistic manner.

IROS Conference 2013 Conference Paper

Acquiring task models for imitation learning through games with a purpose

• Lars Kunze
• Andrei Haidu
• Michael Beetz

Teaching robots everyday tasks like making pancakes by instructions requires interfaces that can be intuitively operated by non-experts. By performing novel manipulation tasks in a virtual environment using a data glove task-related information of the demonstrated actions can directly be accessed and extracted from the simulator. We translate low-level data structures of these simulations into meaningful first-order representations whereby we are able to select data segments and analyze them at an abstract level. Hence, the proposed system is a powerful tool for acquiring examples of manipulation actions and for analyzing them whereby robots can be informed how to perform a task.

IROS Conference 2013 Conference Paper

Automated alignment of specifications of everyday manipulation tasks

• Moritz Tenorth
• Johannes Ziegltrum
• Michael Beetz

Recently, there has been growing interest in enabling robots to use task instructions from the Internet and to share tasks they have learned with each other. To competently use, select and combine such instructions, robots need to be able to find out if different instructions describe the same task, which parts of them are similar and which ones differ. In this paper, we investigate techniques for automatically aligning symbolic task descriptions. We propose to adapt and extend established algorithms for sequence alignment that are commonly used in bioinformatics in order to make them applicable to robot action specifications. The extensions include methods for the comparison of complex sequence elements, for taking the semantic similarity of actions into account, and for aligning descriptions at different levels of granularity. We evaluate the algorithm on two large datasets of observations of human everyday tasks and show that they are able to align action sequences performed by different subjects in very different ways.

IROS Conference 2013 Conference Paper

Decomposing CAD models of objects of daily use and reasoning about their functional parts

• Moritz Tenorth
• Stefan Profanter
• Ferenc Balint-Benczedi
• Michael Beetz

Today's robots are still lacking comprehensive knowledge bases about objects and their properties. Yet, a lot of knowledge is required when performing manipulation tasks to identify abstract concepts like a “handle” or the “blade of a spatula” and to ground them into concrete coordinate frames that can be used to parametrize the robot's actions. In this paper, we present a system that enables robots to use CAD models of objects as a knowledge source and to perform logical inference about object components that have automatically been identified in these models. The system includes several algorithms for mesh segmentation and geometric primitive fitting which are integrated into the robot's knowledge base as procedural attachments to the semantic representation. Bottom-up segmentation methods are complemented by top-down, knowledge-based analysis of the identified components. The evaluation on a diverse set of object models, downloaded from the Internet, shows that the algorithms are able to reliably detect several kinds of object parts.

ICRA Conference 2013 Conference Paper

Fast temporal projection using accurate physics-based geometric reasoning

• Lorenz Mösenlechner
• Michael Beetz

Temporal projection is the computational problem of predicting what will happen when a robot executes its plan. Temporal projection for everyday manipulation tasks such as table setting and cleaning is a challenging task. Symbolic projection methods developed in Artificial Intelligence are too abstract to reason about how to place objects such that they do not hinder future actions. Simulation-based projection is fine-grained enough but computationally too expensive as it is not able to abstract away from the execution of uninteresting actions (such as navigation). In this paper we propose a novel temporal projection mechanism that combines the strengths of both approaches: it is able to abstract away from the execution of continuous but uninteresting actions and provides the realism and fine grainedness needed to reason about critical situations.

IROS Conference 2013 Conference Paper

Interactive environment exploration in clutter

• Megha Gupta
• Thomas Rühr
• Michael Beetz
• Gaurav S. Sukhatme

Robotic environment exploration in cluttered environments is a challenging problem. The number and variety of objects present not only make perception very difficult but also introduce many constraints for robot navigation and manipulation. In this paper, we investigate the idea of exploring a small, bounded environment (e. g. , the shelf of a home refrigerator) by prehensile and non-prehensile manipulation of the objects it contains. The presence of multiple objects results in partial and occluded views of the scene. This inherent uncertainty in the scene's state forces the robot to adopt an observe-plan-act strategy and interleave planning with execution. Objects occupying the space and potentially occluding other hidden objects are rearranged to reveal more of the unseen area. The environment is considered explored when the state (free or occupied) of every voxel in the volume is known. The presented algorithm can be easily adapted to real world problems like object search, taking inventory, and mapping. We evaluate our planner in simulation using various metrics like planning time, number of actions required, and length of planning horizon. We then present an implementation on the PR2 robot and use it for object search in clutter.

ICRA Conference 2013 Conference Paper

Learning probability distributions over partially-ordered human everyday activities

• Moritz Tenorth
• Fernando De la Torre
• Michael Beetz

We propose a method to learn the partially-ordered structure inherent in human everyday activities from observations by exploiting variability in the data. Using statistical relational learning, the system extracts a full-joint probability distribution over the actions that form a task, their (partial) ordering, and their properties. Relevant action properties and relations among actions are learned as those that are consistent among the observations. The models can be used for classifying action sequences, for determining which actions are relevant for a task, which objects are usually manipulated, and which action properties are typical for a person. We evaluate the approach on synthetic data sampled from partial-order trees as well as two real-world data sets of humans activities: the TUM kitchen data set and the CMU MMAC data set. The results show that our approach outperforms sequence-based models like Conditional Random Fields for classifying observations of activities that allow a large amount of variation.

IJCAI Conference 2013 Conference Paper

The RoboEarth Language: Representing and Exchanging Knowledge about Actions, Objects, and Environments (Extended Abstract)

• Moritz Tenorth
• Alexander Perzylo
• Reinhard Lafrenz
• Michael Beetz

The community-based generation of content has been tremendously successful in the World Wide Web – people help each other by providing information that could be useful to others. We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks. RoboEarth is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored. In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.

ICRA Conference 2013 Conference Paper

Tracking-based interactive segmentation of textureless objects

• Karol Hausman
• Ferenc Balint-Benczedi
• Dejan Pangercic
• Zoltán-Csaba Márton
• Ryohei Ueda
• Kei Okada
• Michael Beetz

This paper describes a textureless object segmentation approach for autonomous service robots acting in human living environments. The proposed system allows a robot to effectively segment textureless objects in cluttered scenes by leveraging its manipulation capabilities. In our pipeline, the cluttered scenes are first statically segmented using state-of-the-art classification algorithm and then the interactive segmentation is deployed in order to resolve this possibly ambiguous static segmentation. In the second step the RGBD (RGB + Depth) sparse features, estimated on the RGBD point cloud from the Kinect sensor, are extracted and tracked while motion is induced into a scene. Using the resulting feature poses, the features are then assigned to their corresponding objects by means of a graph-based clustering algorithm. In the final step, we reconstruct the dense models of the objects from the previously clustered sparse RGBD features. We evaluated the approach on a set of scenes which consist of various textureless flat (e. g. box-like) and round (e. g. cylinder-like) objects and the combinations thereof.

ICRA Conference 2012 Conference Paper

A generalized framework for opening doors and drawers in kitchen environments

• Thomas Rühr
• Jürgen Sturm
• Dejan Pangercic
• Michael Beetz
• Daniel Cremers

In this paper, we present a generalized framework for robustly operating previously unknown cabinets in kitchen environments. Our framework consists of the following four components: (1) a module for detecting both Lambertian and non-Lambertian (i. e. specular) handles, (2) a module for opening and closing novel cabinets using impedance control and for learning their kinematic models, (3) a module for storing and retrieving information about these objects in the map, and (4) a module for reliably operating cabinets of which the kinematic model is known. The presented work is the result of a collaboration of three PR2 beta sites. We rigorously evaluated our approach on 29 cabinets in five real kitchens located at our institutions. These kitchens contained 13 drawers, 12 doors, 2 refrigerators and 2 dishwashers. We evaluated the overall performance of detecting the handle of a novel cabinet, operating it and storing its model in a semantic map. We found that our approach was successful in 51. 9% of all 104 trials. With this work, we contribute a well-tested building block of open-source software for future robotic service applications.

IROS Conference 2012 Conference Paper

A unified representation for reasoning about robot actions, processes, and their effects on objects

• Moritz Tenorth
• Michael Beetz

Mobile manipulation robots are becoming more and more common and begin to extend their task spectrum towards more general housework activities. The sequence of actions needed to accomplish such tasks can be obtained from instructions on the Internet originally written for humans. While giving valuable information about the types of actions and some of their parameters, these instructions usually lack information that humans consider to be obvious. In this paper, we investigate how we can equip robots with sufficient knowledge and inference mechanisms to competently detect and fill such knowledge gaps in descriptions of everyday activities. We present methods for projecting the effects of actions and processes, for inferring action parameters like the objects and locations to be used, and introduce representations for reasoning about object transformations resulting from the effects of actions.

IROS Conference 2012 Conference Paper

Everything robots always wanted to know about housework (but were afraid to ask)

• Daniel Nyga
• Michael Beetz

In this paper we discuss the problem of action-specific knowledge processing, representation and acquisition by autonomous robots performing everyday activities. We report on a thorough analysis of the household domain, which has been performed on a large corpus of natural-language instructions from the Web and underlines the supreme need of action-specific knowledge for robots acting in those environments. We introduce the concept of Probabilistic Robot Action Cores (PRAC) that are well-suited for encoding such knowledge in a probabilistic first-order knowledge base. We additionally show how such a knowledge base can be acquired by natural language and we address the problems of incompleteness, underspecification and ambiguity of naturalistic action specifications and point out how PRAC models can tackle those.

IROS Conference 2012 Conference Paper

Improving robot manipulation through fingertip perception

• Alexis Maldonado
• Humberto Alvarez
• Michael Beetz

Better sensing is crucial to improve robotic grasping and manipulation. Most robots currently have very limited perception in their manipulators, typically only fingertip position and velocity. Additional sensors make richer interactions with the objects possible. In this paper, we present a versatile, robust and low cost sensor for robot fingertips, that can improve robotic grasping and manipulation in several ways: 3D reconstruction of the shape of objects, material surface classification, and object slip detection. We extended TUM-Rosie, our robot for mobile manipulation, with fingertip sensors on its humanoid robotic hand, and show the advantages of the fingertip sensor integrated in our robot system.

ICRA Conference 2012 Conference Paper

Learning organizational principles in human environments

• Martin J. Schuster
• Dominik Jain
• Moritz Tenorth
• Michael Beetz

In the context of robotic assistants in human everyday environments, pick and place tasks are beginning to be competently solved at the technical level. The question of where to place objects or where to pick them up from, among other higher-level reasoning tasks, is therefore gaining practical relevance. In this work, we consider the problem of identifying the organizational structure within an environment, i. e. the problem of determining organizational principles that would allow a robot to infer where to best place a particular, previously unseen object or where to reasonably search for a particular type of object given past observations about the allocation of objects to locations in the environment. This problem can be reasonably formulated as a classification task. We claim that organizational principles are governed by the notion of similarity and provide an empirical analysis of the importance of various features in datasets describing the organizational structure of kitchens. For the aforementioned classification tasks, we compare standard classification methods, reaching average accuracies of at least 79% in all scenarios. We thereby show that, in particular, ontology-based similarity measures are well-suited as highly discriminative features. We demonstrate the use of learned models of organizational principles in a kitchen environment on a real robot system, where the robot identifies a newly acquired item, determines a suitable location and then stores the item accordingly.

ICRA Conference 2012 Conference Paper

Movement-aware action control - Integrating symbolic and control-theoretic action execution

• Ingo Kresse
• Michael Beetz

In this paper we propose a bridge between a symbolic reasoning system and a task function based controller. We suggest to use modular position- and force constraints, which are represented as action-object-object triples on the symbolic side and as task function parameters on the controller side. This description is a considerably more fine-grained interface than what has been seen in high-level robot control systems before. It can preserve the 'null space' of the task and make it available to the control level. We demonstrate how a symbolic description can be translated to a control-level description that is executable on the robot. We describe the relation to existing robot knowledge bases and indicate information sources for generating constraints on the symbolic side. On the control side we then show how our approach outperforms a traditional controller, by exploiting the task's null space, leading to a significantly extended work space.

ICRA Conference 2012 Conference Paper

Real-time compression of point cloud streams

• Julius Kammerl
• Nico Blodow
• Radu Bogdan Rusu
• Suat Gedikli
• Michael Beetz
• Eckehard G. Steinbach

We present a novel lossy compression approach for point cloud streams which exploits spatial and temporal redundancy within the point data. Our proposed compression framework can handle general point cloud streams of arbitrary and varying size, point order and point density. Furthermore, it allows for controlling coding complexity and coding precision. To compress the point clouds, we perform a spatial decomposition based on octree data structures. Additionally, we present a technique for comparing the octree data structures of consecutive point clouds. By encoding their structural differences, we can successively extend the point clouds at the decoder. In this way, we are able to detect and remove temporal redundancy from the point cloud data stream. Our experimental results show a strong compression performance of a ratio of 14 at 1 mm coordinate precision and up to 40 at a coordinate precision of 9 mm.

ICRA Conference 2012 Conference Paper

Robots that validate learned perceptual models

• Ulrich Klank
• Lorenz Mösenlechner
• Alexis Maldonado
• Michael Beetz

Service robots that should operate autonomously need to perform actions reliably, and be able to adapt to their changing environment using learning mechanisms. Optimally, robots should learn continuously but this approach often suffers from problems like over-fitting, drifting or dealing with incomplete data. In this paper, we propose a method to automatically validate autonomously acquired perception models. These perception models are used to localize objects in the environment with the intention of manipulating them with the robot. Our approach verifies the learned perception models by moving the robot, trying to re-detect an object and then to grasp it. From observable failures of these actions and highlevel loop-closures to validate the eventual success, we can derive certain qualities of our models and our environment. We evaluate our approach by using two different detection algorithms, one using 2D RGB data and one using 3D point clouds. We show that our system is able to improve the perception performance significantly by learning which of the models is better in a certain situation and a specific context. We show how additional validation allows for successful continuous learning. The strictest precondition for learning such perceptual models is correct segmentation of objects which is evaluated in a second experiment.

ICRA Conference 2012 Conference Paper

Searching objects in large-scale indoor environments: A decision-theoretic approach

• Lars Kunze
• Michael Beetz
• Manabu Saito
• Haseru Azuma
• Kei Okada
• Masayuki Inaba

Many of today's mobile robots are supposed to perform everyday manipulation tasks autonomously. However, in large-scale environments, a task-related object might be out of the robot's reach. Hence, the robot first has to search for the object in its environment before it can perform the task. In this paper, we present a decision-theoretic approach for searching objects in large-scale environments using probabilistic environment models and utilities associated with object locations. We demonstrate the feasibility of our approach by integrating it into a robot system and by conducting experiments where the robot is supposed to search different objects with various strategies in the context of fetch-and-delivery tasks within a multi-level building.

IROS Conference 2012 Conference Paper

Semantic Object Maps for robotic housework - representation, acquisition and use

• Dejan Pangercic
• Benjamin Pitzer
• Moritz Tenorth
• Michael Beetz

In this article we investigate the representation and acquisition of Semantic Objects Maps (SOMs) that can serve as information resources for autonomous service robots performing everyday manipulation tasks in kitchen environments. These maps provide the robot with information about its operation environment that enable it to perform fetch and place tasks more efficiently and reliably. To this end, the semantic object maps can answer queries such as the following ones: “What do parts of the kitchen look like? ”, “How can a container be opened and closed? ”, “Where do objects of daily use belong? ”, “What is inside of cupboards/drawers? ”, etc. The semantic object maps presented in this article, which we call SOM +, extend the first generation of SOMs presented by Rusu et al. [1] in that the representation of SOM + is designed more thoroughly and that SOM + also include knowledge about the appearance and articulation of furniture objects. Also, the acquisition methods for SOM + substantially advance those developed in [1] in that SOM + are acquired autonomously and with low-cost (Kinect) instead of very accurate (laser-based) 3D sensors. In addition, perception methods are more general and are demonstrated to work in different kitchen environments.

ICRA Conference 2012 Conference Paper

The RoboEarth language: Representing and exchanging knowledge about actions, objects, and environments

• Moritz Tenorth
• Alexander Perzylo
• Reinhard Lafrenz
• Michael Beetz

The community-based generation of content has been tremendously successful in the World Wide Web - people help each other by providing information that could be useful to others. We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks. RoboEarth is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored. In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.

IROS Conference 2011 Conference Paper

Autonomous semantic mapping for robots performing everyday manipulation tasks in kitchen environments

• Nico Blodow
• Lucian Cosmin Goron
• Zoltán-Csaba Márton
• Dejan Pangercic
• Thomas Rühr
• Moritz Tenorth
• Michael Beetz

In this work we report about our efforts to equip service robots with the capability to acquire 3D semantic maps. The robot autonomously explores indoor environments through the calculation of next best view poses, from which it assembles point clouds containing spatial and registered visual information. We apply various segmentation methods in order to generate initial hypotheses for furniture drawers and doors. The acquisition of the final semantic map makes use of the robot's proprioceptive capabilities and is carried out through the robot's interaction with the environment. We evaluated the proposed integrated approach in the real kitchen in our laboratory by measuring the quality of the generated map in terms of the map's applicability for the task at hand (e. g. resolving counter candidates by our knowledge processing system).

ICRA Conference 2011 Conference Paper

How-models of human reaching movements in the context of everyday manipulation activities

• Daniel Nyga
• Moritz Tenorth
• Michael Beetz

We present a system for learning models of human reaching trajectories in the context of everyday manipulation activities. Different kinds of trajectories are automatically discovered, and each of them is described by its semantic context. In a first step, the system clusters trajectories in observations of human everyday activities based on their shapes, and then learns the relation between these trajectories and the contexts in which they are used. The resulting models can be used for robots to select a trajectory to use in a given context. They can also serve as powerful prediction models for human motions to improve human-robot interaction. Experiments on the TUM kitchen data set show that the method is capable of discovering meaningful clusters in real-world observations of everyday activities like setting a table.

IROS Conference 2011 Conference Paper

Logic programming with simulation-based temporal projection for everyday robot object manipulation

• Lars Kunze
• Mihai Emanuel Dolha
• Michael Beetz

In everyday object manipulation tasks, like making a pancake, autonomous robots are required to decide on the appropriate action parametrizations in order to achieve desired (and to avoid undesired) outcomes. For determining the right parameters for actions like pouring a pancake mix onto a pancake maker, robots need capabilities to predict the physical consequences of their own manipulation actions. In this work, we integrate a simulation-based approach for making temporal projections for robot manipulation actions into the logic programming language PROLOG. The realized system enables robots to determine action parameters that bring about certain effects by utilizing simulation-based temporal projections within PROLOG's chronological backtracking mechanism. For a set of formal parameters and their respective ranges of values, the developed system translates the manipulation problems into physical simulations, monitors and logs the relevant data structures of the simulations, translates the logged data back into first-order time-interval-based representations, called timelines, and eventually evaluates the individual timelines with respect to specified performance criteria. Integrating the proposed approach into robot control programs allow robots to mentally simulate the consequences of different action parametrizations before committing to them and thereby to reduce the number of undesired outcomes.

IROS Conference 2011 Conference Paper

Parameterizing actions to have the appropriate effects

• Lorenz Mösenlechner
• Michael Beetz

Robots that are to perform their tasks reliably and skillfully in complex domains such as a human household need to apply both, qualitative and quantitative reasoning to achieve their goals. Consider a robot whose task is to make pancakes, and part of the plan is to put down the bottle with pancake mix after pouring it on the pan. The put-down location of the bottle is heavily under-specified but has a critical influence on the overall performance of the plan. For instance, when it places it at a location where it occludes other objects, the robot cannot see and grasp the occluded objects anymore unless the bottle is removed again. Other important aspects include stability and reachability. Objects should not flip over or fall. A badly chosen put-down location can “block” trajectories for grasping other objects that were valid before and can even prevent the robot from reaching these objects. In this paper, we show a lightweight and fast reasoning system that integrates qualitative and quantitative reasoning based on Prolog. We demonstrate how we implement predicates that make use of OpenGL, the Bullet physics engine and inverse kinematics calculation. Equipped with generative models yielding pose candidates, our system allows for the generation of action parameters such as put down locations under the constraints of the current and future actions in real time.

AAMAS Conference 2011 Conference Paper

Simulation-based Temporal Projection of Everyday Robot Object Manipulation

• Lars Kunze
• Mihai Emanuel Dolha
• Emitza Guzman
• Michael Beetz

Performing everyday manipulation tasks successfully depends on the ability of autonomous robots to appropriately account for the physical behavior of task-related objects. Meaning that robots have to predict and consider the physical effects of their possible actions to take. In this work we investigate a simulation-based approach to naive physics temporal projection in the context of autonomous robot everyday manipulation. We identify the abstractions underlying typical first-order axiomatizations as the key obstacles for making valid naive physics predictions. We propose that temporal projection for naive physics problems should not be performed based on abstractions but rather based on detailed physical simulations. This idea is realized as a temporal projection system for autonomous manipulation robots that translates naive physics problems into parametrized physical simulation tasks, that logs the data structures and states traversed in simulation, and translates the logged data back into symbolic time-interval-based first-order representations. Within this paper, we describe the concept and implementation of the temporal projection system and present the example of an egg-cracking robot for demonstrating its feasibility.

ICRA Conference 2011 Conference Paper

Towards semantic robot description languages

• Lars Kunze
• Tobias Roehm
• Michael Beetz

There is a semantic gap between simple but high-level action instructions like “Pick up the cup with the right hand” and low-level robot descriptions that model, for example, the structure and kinematics of a robot's manipulator. Currently, programmers bridge this gap by mapping abstract instructions to parametrized algorithms and rigid body parts of a robot within their control programs. By linking descriptions of robot components, i. e. sensors, actuators and control programs, via capabilities to actions in an ontology we equip robots with knowledge about themselves that allows them to infer the required components for performing a given action. Thereby a robot that is instructed by an end-user, a programmer, or even another robot to perform a certain action, can assess itself whether it is able and how to perform the requested action. This self-knowledge for robots could considerably change the way of robot control, robot interaction, robot programming, and multi-robot communication.

ICRA Conference 2011 Conference Paper

Transparent object detection and reconstruction on a mobile platform

• Ulrich Klank
• Daniel Carton
• Michael Beetz

In this paper we propose a novel approach to detect and reconstruct transparent objects. This approach makes use of the fact that many transparent objects, especially the ones consisting of usual glass, absorb light in certain wavelengths [1]. Given a controlled illumination, this absorption is measurable in the intensity response by comparison to the background. We show the usage of a standard infrared emitter and the intensity sensor of a time of flight (ToF) camera to reconstruct the structure given we have a second view point. The structure can not be measured by the usual 3D measurements of the ToF camera. We take advantage of this fact by deriving this internal sensory contradiction from two ToF images and reconstruct an approximated surface of the original transparent object. Therefor we are using a perspectively invariant matching in the intensity channels from the first to the second view of initially acquired candidates. For each matched pixel in the first view a 3D movement can be predicted given their original 3D measurement and the known distance to the second camera position. If their line of sight did not pass a transparent object or suffered any other major defect, this prediction will highly correspond to the actual measured 3D points of the second view. Otherwise, if a detectable error occurs, we approximate a more exact point to point matching and reconstruct the original shape by triangulating the points in the stereo setup. We tested our approach using a mobile platform with one Swissranger SR4k. As this platform is mobile, we were able to create a stereo setup by moving it. Our results show a detection of transparent objects on tables while simultaneously identifying opaque objects that also existed in the test setup. The viability of our results is demonstrated by a successful automated manipulation of the respective transparent object.

ECAI Conference 2010 Conference Paper

Adaptive Markov Logic Networks: Learning Statistical Relational Models with Dynamic Parameters

• Dominik Jain
• Andreas Barthels
• Michael Beetz

Statistical relational models, such as Markov logic networks, seek to compactly describe properties of relational domains by representing general principles about objects belonging to particular classes. Models are intended to be independent of the set of objects to which these principles can be applied, and it is assumed that the principles will soundly generalize across arbitrary sets of objects. In this paper, we point out limitations of models that seek to represent the corresponding principles with a fixed set of parameters and discuss the conditions under which the soundness of fixed parameters is indeed questionable. We propose a novel representation formalism called adaptive Markov logic networks to allow more flexible representations of relational domains, which involve parameters that are dynamically adjusted to fit the properties of an instantiation by phrasing the model's parameters as functions over attributes of the instantiation at hand. We empirically demonstrate the value of our learning and representation system on a simple but well-motivated example domain.

IROS Conference 2010 Conference Paper

Becoming action-aware through reasoning about logged plan execution traces

• Lorenz Mösenlechner
• Nikolaus Demmel
• Michael Beetz

Robots that know what they are doing can solve their tasks more reliably, flexibly, and efficiently. They can even explain what they were doing, how and why. In this paper we describe a system that not only is capable of executing flexible and reliable plans on a robotic platform but can also explain control decisions and the reason for specific actions, diagnose the cause of failures and answer queries about the robot's beliefs. For instance, when queried why it opened the cupboard door, the robot might answer that it did so because it believed Michael's cup to be in there. This type of reasoning is not only helpful for debugging but also provides the mechanisms for complex monitoring and failure handling that is not based on local failures and exception handling but on the expressive formulation of error patterns in first order logics. Our system is based on semantic annotations of plans, a fast logging mechanism and the computation of predicates in a first-order representation based on the execution trace.

IROS Conference 2010 Conference Paper

Combining perception and knowledge processing for everyday manipulation

• Dejan Pangercic
• Moritz Tenorth
• Dominik Jain
• Michael Beetz

This paper describes and discusses the K-COPMAN (Knowledge-enabled Cognitive Perception for Manipulation) system, which enables autonomous robots to generate symbolic representations of perceived objects and scenes and to infer answers to complex queries that require the combination of perception and knowledge processing. Using K-COPMAN, the robot can solve inference tasks such as identifying items that are likely to be missing on a breakfast table. To the programmer K-COPMAN, is presented as a logic programming system that can be queried just like a symbolic knowledge base. Internally, K-COPMAN is realized through a data structure framework together with a library of state-of-the-art perception mechanisms for mobile manipulation in human environments. Key features of K-COPMAN are that it can make a robot environment-aware and that it supports goal-directed as well as passive perceptual processing. K-COPMAN is fully integrated into an autonomous mobile manipulation robot and is realized within the open-source robot library ROS.

IROS Conference 2010 Conference Paper

CRAM - A Cognitive Robot Abstract Machine for everyday manipulation in human environments

• Michael Beetz
• Lorenz Mösenlechner
• Moritz Tenorth

This paper describes CRAM (Cognitive Robot Abstract Machine) as a software toolbox for the design, the implementation, and the deployment of cognition-enabled autonomous robots performing everyday manipulation activities. CRAM equips autonomous robots with lightweight reasoning mechanisms that can infer control decisions rather than requiring the decisions to be preprogrammed. This way CRAM-programmed autonomous robots are much more flexible, reliable, and general than control programs that lack such cognitive capabilities. CRAM does not require the whole domain to be stated explicitly in an abstract knowledge base. Rather, it grounds symbolic expressions in the knowledge representation into the perception and actuation routines and into the essential data structures of the control programs. In the accompanying video, we show complex mobile manipulation tasks performed by our household robot that were realized using the CRAM infrastructure.

IROS Conference 2010 Conference Paper

General 3D modelling of novel objects from a single view

• Zoltán-Csaba Márton
• Dejan Pangercic
• Nico Blodow
• Jonathan Kleinehellefort
• Michael Beetz

In this paper we present a method for building models for grasping from a single 3D snapshot of a scene composed of objects of daily use in human living environments. We employ fast shape estimation, probabilistic model fitting and verification methods capable of dealing with different kinds of symmetries, and combine these with a triangular mesh of the parts that have no other representation to model previously unseen objects of arbitrary shape. Our approach is enhanced by the information given by the geometric clues about different parts of objects which serve as prior information for the selection of the appropriate reconstruction method. While we designed our system for grasping based on single view 3D data, its generality allows us to also use the combination of multiple views. We present two application scenarios that require complete geometric models: grasp planning and locating objects in camera images.

IROS Conference 2010 Conference Paper

ORO, a knowledge management platform for cognitive architectures in robotics

• Séverin Lemaignan
• Raquel Ros
• Lorenz Mösenlechner
• Rachid Alami 0001
• Michael Beetz

This paper presents an embeddable knowledge processing framework, along with a common-sense ontology, designed for robotics. We believe that a direct and explicit integration of cognition is a compulsory step to enable human-robots interaction in semantic-rich human environments like our houses. The OpenRobots Ontology (ORO) kernel allows to turn previously acquired symbols into concepts linked to each other. It enables in turn reasoning and the implementation of other advanced cognitive functions like events, categorization, memory management and reasoning on parallel cognitive models. We validate this framework on several cognitive scenarii that have been implemented on three different robotic architectures.

IROS Conference 2010 Conference Paper

Prediction of action outcomes using an object model

• Federico Ruiz-Ugalde
• Gordon Cheng
• Michael Beetz

When a robot wants to manipulate an object, it needs to know what action to execute to obtain the desired result. In most of the cases, the actions that can be applied to an object consist of exerting forces to it. If a robot is able to predict what will happen to an object when some force is applied to it, then it's possible to build a controller that solves the inverse problem of what force needs to be applied in order to get a desired result. To accomplish this, the first task is to build an object model and second to get the right parameters for it. The goals of this paper are 1) to demonstrate the use of an object model to predict outcomes of actions, and 2) to adapt this model to an specific object instance for a specific robot.

ICRA Conference 2010 Conference Paper

Priming transformational planning with observations of human activities

• Moritz Tenorth
• Michael Beetz

People perform daily activities in many different ways. When setting a table, they might use a tray, stack plates, stack cups on plates, leave the doors of a cupboard open when taking several items out of it. Similarly flexible behavior is desired when mobile robots perform household tasks. Moreover, they should perform actions in a way that they are accepted by the people, for example by showing human-like behavior. In this paper we propose to extend a transformational planning system with models characterizing the behavior produced by the different plans in the plan library. These models are used by the robot to select a plan that resembles human behavior. In addition to acting more human-like, this helps the robot choose good plans for a task by imitating humans instead of performing exhaustive search. We show the feasibility of this approach using a household robot application as an example and present empirical results on the classification accuracy in this domain.

IROS Conference 2010 Conference Paper

Robotic grasping of unmodeled objects using time-of-flight range data and finger torque information

• Alexis Maldonado
• Ulrich Klank
• Michael Beetz

Robotic grasping in an open environment requires both object-specific as well as general grasping skills. When the objects are previously known it is possible to employ techniques that exploit object models, like geometrical grasping simulators. On the other hand, a competent system will also be able to deal with unmodeled objects using general solutions. In this paper we present an integrated system for autonomous rigid-object pick-up tasks in domestic environments, focusing on the gripping of unmodeled objects and exploiting sensor feedback from the robot hand to monitor the grasp. We describe the perception system based on time-of-flight range data, the grasp pose optimization algorithm and the grasp execution. The performance and robustness of the system is validated by experiments including pick-up tasks on many different common kitchen items.

ICRA Conference 2010 Conference Paper

Understanding and executing instructions for everyday manipulation tasks from the World Wide Web

• Moritz Tenorth
• Daniel Nyga
• Michael Beetz

Service robots will have to accomplish more and more complex, open-ended tasks and regularly acquire new skills. In this work, we propose a new approach to the problem of generating plans for such household robots. Instead composing them from atomic actions - the common approach in robot planning - we propose to transform task descriptions on web sites like ehow.com into executable robot plans. We present methods for automatically converting the instructions from natural language into a formal, logic-based representation, for resolving the word senses using the WordNet database and the Cyc ontology, and for exporting the generated plans into the mobile robot's plan language RPL. We discuss the problem of inferring information that is missing in these descriptions and the problem of grounding the abstract task descriptions in the perception and action system, and we propose techniques for solving them. The whole system works autonomously without human interaction. It has successfully been tested with a set of about 150 natural language directives, of which up to 80% could be correctly transformed.

ICRA Conference 2009 Conference Paper

3D model selection from an internet database for robotic vision

• Ulrich Klank
• M. Zeeshan Zia
• Michael Beetz

We propose a new method for automatically accessing an internet database of 3D models that are searchable only by their user-annotated labels, for using them for vision and robotic manipulation purposes. Instead of having only a local database containing already seen objects, we want to use shared databases available over the internet. This approach while having the potential to dramatically increase the visual recognition capability of robots, also poses certain problems, like wrong annotation due to the open nature of the database, or overwhelming amounts of data (many 3D models) or the lack of relevant data (no models matching a specified label). To solve those problems we propose the following: First, we present an outlier/inlier classification method for reducing the number of results and discarding invalid 3D models that do not match our query. Second, we utilize an approach from computer graphics, the so called ‘morphing’, to this application to specialize the models, in order to describe more objects. Third, we search for 3D models using a restricted search space, as obtained from our knowledge of the environment. We show our classification and matching results and finally show how we can recover the correct scaling with the stereo setup of our robot.

IROS Conference 2009 Conference Paper

Action-related place-based mobile manipulation

• Freek Stulp
• Andreas Fedrizzi
• Michael Beetz

In mobile manipulation, the position to which the robot navigates has a large influence on the ease with which a subsequent manipulation action can be performed. Whether a manipulation action succeeds depends on many factors, such as the robot's hardware configuration, the controllers the robot uses to achieve navigation and manipulation, the task context, and uncertainties in state estimation. In this paper, we present ‘ARPLACE’, an action-related place which takes these factors, and the context in which the actions are performed into account. Through experience-based learning, the robot first learns a so-called generalized success model, which discerns between positions from which manipulation succeeds or fails. On-line, this model is used to compute a ARPLACE, a probability distribution that maps positions to a predicted probability of successful manipulation, and takes the uncertainty in the robot and object's position into account. In an empirical evaluation, we demonstrate that using ARPLACEs for least-commitment navigation improves the success rate of subsequent manipulation tasks substantially.

IROS Conference 2009 Conference Paper

Close-range scene segmentation and reconstruction of 3D point cloud maps for mobile manipulation in domestic environments

• Radu Bogdan Rusu
• Nico Blodow
• Zoltán-Csaba Márton
• Michael Beetz

In this paper we present a framework for 3D geometric shape segmentation for close-range scenes used in mobile manipulation and grasping, out of sensed point cloud data. Our proposed approach proposes a robust geometric mapping pipeline for large input datasets that extracts relevant objects useful for a personal robotic assistant to perform manipulation tasks. The objects are segmented out from partial views and a reconstructed model is computed by fitting geometric primitive classes such as planes, spheres, cylinders, and cones. The geometric shape coefficients are then used to reconstruct missing data. Residual points are resampled and triangulated, to create smooth decoupled surfaces that can be manipulated. The resulted map is represented as a hybrid concept and is comprised of 3D shape coefficients and triangular meshes used for collision avoidance in manipulation routines.

ICRA Conference 2009 Conference Paper

Equipping robot control programs with first-order probabilistic reasoning capabilities

• Dominik Jain
• Lorenz Mösenlechner
• Michael Beetz

An autonomous robot system that is to act in a real-world environment is faced with the problem of having to deal with a high degree of both complexity as well as uncertainty. Therefore, robots should be equipped with a knowledge representation system that is able to soundly handle both aspects. In this paper, we thus introduce an architecture that provides a coupling between plan-based robot controllers and a probabilistic knowledge representation system based on recent developments in statistical relational learning, which possesses the required level of expressiveness and generality. We outline possible applications of the corresponding models in the context of robot control, discussing suitable representation formalisms, inference and learning methods as well as transparent extensions of a robot planning language that allow robot control programs to soundly integrate the results of probabilistic inference into their plan generation process.

IROS Conference 2009 Conference Paper

Fast geometric point labeling using conditional random fields

• Radu Bogdan Rusu
• Andreas Holzbach
• Nico Blodow
• Michael Beetz

In this paper we present a new approach for labeling 3D points with different geometric surface primitives using a novel feature descriptor - the Fast Point Feature Histograms, and discriminative graphical models. To build informative and robust 3D feature point representations, our descriptors encode the underlying surface geometry around a point p using multi-value histograms. This highly dimensional feature space copes well with noisy sensor data and is not dependent on pose or sampling density. By defining classes of 3D geometric surfaces and making use of contextual information using Conditional Random Fields (CRFs), our system is able to successfully segment and label 3D point clouds, based on the type of surfaces the points are lying on. We validate and demonstrate the method's efficiency by comparing it against similar initiatives as well as present results for table setting datasets acquired in indoor environments.

ICRA Conference 2009 Conference Paper

Fast Point Feature Histograms (FPFH) for 3D registration

• Radu Bogdan Rusu
• Nico Blodow
• Michael Beetz

In our recent work [1], [2], we proposed Point Feature Histograms (PFH) as robust multi-dimensional features which describe the local geometry around a point p for 3D point cloud datasets. In this paper, we modify their mathematical expressions and perform a rigorous analysis on their robustness and complexity for the problem of 3D registration for overlapping point cloud views. More concretely, we present several optimizations that reduce their computation times drastically by either caching previously computed values or by revising their theoretical formulations. The latter results in a new type of local features, called Fast Point Feature Histograms (FPFH), which retain most of the discriminative power of the PFH. Moreover, we propose an algorithm for the online computation of FPFH features for realtime applications. To validate our results we demonstrate their efficiency for 3D registration and propose a new sample consensus based method for bringing two datasets into the convergence basin of a local non-linear optimizer: SAC-IA (SAmple Consensus Initial Alignment).

IROS Conference 2009 Conference Paper

KNOWROB - knowledge processing for autonomous personal robots

• Moritz Tenorth
• Michael Beetz

Knowledge processing is an essential technique for enabling autonomous robots to do the right thing to the right object in the right way. Using knowledge processing the robots can achieve more flexible and general behavior and better performance. While knowledge representation and reasoning has been a well-established research field in artificial intelligence for several decades, little work has been done to design and realize knowledge processing mechanisms for the use in the context of robotic control. In this paper, we report on KNOWROB, a knowledge processing system particularly designed for autonomous personal robots. KNOWROB is a first-order knowledge representation based on description logics that provides specific mechanisms and tools for action-centered representation, for the automated acquisition of grounded concepts through observation and experience, for reasoning about and managing uncertainty, and for fast inference - knowledge processing features that are particularly necessary for autonomous robot control.

ICRA Conference 2009 Conference Paper

Leaving Flatland: Toward real-time 3D navigation

• Benoit Morisset
• Radu Bogdan Rusu
• Aravind Sundaresan
• Kris Hauser
• Motilal Agrawal
• Jean-Claude Latombe
• Michael Beetz

We report our first experiences with Leaving Flatland, an exploratory project that studies the key challenges of closing the loop between autonomous perception and action on challenging terrain. We propose a comprehensive system for localization, mapping, and planning for the RHex mobile robot in fully 3D indoor and outdoor environments. This system integrates Visual Odometry-based localization with new techniques in real-time 3D mapping from stereo data. The motion planner uses a new decomposition approach to adapt existing 2D planning techniques to operate in 3D terrain. We test the map-building and motion-planning subsystems on real and synthetic data, and show that they have favorable computational performance for use in high-speed autonomous navigation.

IROS Conference 2009 Conference Paper

Model-based and learned semantic object labeling in 3D point cloud maps of kitchen environments

• Radu Bogdan Rusu
• Zoltán-Csaba Márton
• Nico Blodow
• Andreas Holzbach
• Michael Beetz

We report on our experiences regarding the acquisition of hybrid Semantic 3D Object Maps for indoor household environments, in particular kitchens, out of sensed 3D point cloud data. Our proposed approach includes a processing pipeline, including geometric mapping and learning, for processing large input datasets and for extracting relevant objects useful for a personal robotic assistant to perform complex manipulation tasks. The type of objects modeled are objects which perform utilitarian functions in the environment such as kitchen appliances, cupboards, tables, and drawers. The resulted model is accurate enough to use it in physics-based simulations, where doors of 3D containers can be opened based on their hinge position. The resulted map is represented as a hybrid concept and is comprised of both the hierarchically classified objects and triangular meshes used for collision avoidance in manipulation routines.

ICRA Conference 2009 Conference Paper

On fast surface reconstruction methods for large and noisy point clouds

• Zoltán-Csaba Márton
• Radu Bogdan Rusu
• Michael Beetz

In this paper we present a method for fast surface reconstruction from large noisy datasets. Given an unorganized 3D point cloud, our algorithm recreates the underlying surface's geometrical properties using data resampling and a robust triangulation algorithm in near realtime. For resulting smooth surfaces, the data is resampled with variable densities according to previously estimated surface curvatures. Incremental scans are easily incorporated into an existing surface mesh, by determining the respective overlapping area and reconstructing only the updated part of the surface mesh. The proposed framework is flexible enough to be integrated with additional point label information, where groups of points sharing the same label are clustered together and can be reconstructed separately, thus allowing fast updates via triangular mesh decoupling. To validate our approach, we present results obtained from laser scans acquired in both indoor and outdoor environments.

IROS Conference 2009 Conference Paper

Probabilistic categorization of kitchen objects in table settings with a composite sensor

• Zoltán-Csaba Márton
• Radu Bogdan Rusu
• Dominik Jain
• Ulrich Klank
• Michael Beetz

In this paper, we investigate the problem of 3D object categorization of objects typically present in kitchen environments, from data acquired using a composite sensor. Our framework combines different sensing modalities and defines descriptive features in various spaces for the purpose of learning good object models. By fusing the 3D information acquired from a composite sensor that includes a color stereo camera, a time-of-flight (TOF) camera, and a thermal camera, we augment 3D depth data with color and temperature information which helps disambiguate the object categorization process. We make use of statistical relational learning methods (Markov Logic Networks and Bayesian Logic Networks) to capture complex interactions between the different feature spaces. To show the effectiveness of our approach, we analyze and validate the proposed system for the problem of recognizing objects in table settings scenarios.

IROS Conference 2009 Conference Paper

Real-time perception-guided motion planning for a personal robot

• Radu Bogdan Rusu
• Ioan Alexandru Sucan
• Brian P. Gerkey
• Sachin Chitta
• Michael Beetz
• Lydia E. Kavraki

This paper presents significant steps towards the online integration of 3D perception and manipulation for personal robotics applications. We propose a modular and distributed architecture, which seamlessly integrates the creation of 3D maps for collision detection and semantic annotations, with a real-time motion replanning framework. To validate our system, we present results obtained during a comprehensive mobile manipulation scenario, which includes the fusion of the above components with a higher level executive.

ICAPS Conference 2009 Conference Paper

Using Physics- and Sensor-based Simulation for High-Fidelity Temporal Projection of Realistic Robot Behavior

• Lorenz Mösenlechner
• Michael Beetz

Planning means deciding on the future course of action based on predictions of what will happen when an activity is carried out in one way or the other. As we apply action planning to autonomous, sensor-guided mobile robots with manipulators or even to humanoid robots we need very realistic and detailed predictions of the behavior generated by a plan in order to improve the robot's performance substantially. In this paper we investigate the high-fidelity temporal projection of realistic robot behavior based on physics- and sensor-based simulation systems. We equip a simulator and interpreter with means to log simulated plan executions into a database. A logic-based query and inference mechanism then retrieves and reconstructs the necessary information from the database and translates the information into a first-order representation of robot plans and the behavior they generate. The query language enables the robot planning system to infer the intentions, the beliefs, and the world state at any projected time. It also allows the planning system to recognize, diagnose, and analyze various plan failures typical for performing everyday manipulation tasks.

IROS Conference 2008 Conference Paper

Aligning point cloud views using persistent feature histograms

• Radu Bogdan Rusu
• Nico Blodow
• Zoltán-Csaba Márton
• Michael Beetz

In this paper we investigate the usage of persistent point feature histograms for the problem of aligning point cloud data views into a consistent global model. Given a collection of noisy point clouds, our algorithm estimates a set of robust 16D features which describe the geometry of each point locally. By analyzing the persistence of the features at different scales, we extract an optimal set which best characterizes a given point cloud. The resulted persistent features are used in an initial alignment algorithm to estimate a rigid transformation that approximately registers the input datasets. The algorithm provides good starting points for iterative registration algorithms such as ICP (Iterative Closest Point), by transforming the datasets to its convergence basin. We show that our approach is invariant to pose and sampling density, and can cope well with noisy data coming from both indoor and outdoor laser scans.

IROS Conference 2008 Conference Paper

Functional object mapping of kitchen environments

• Radu Bogdan Rusu
• Zoltán-Csaba Márton
• Nico Blodow
• Mihai Emanuel Dolha
• Michael Beetz

In this paper we investigate the acquisition of 3D functional object maps for indoor household environments, in particular kitchens, out of 3D point cloud data. By modeling the static objects in the world into hierarchical classes in the map, such as cupboards, tables, drawers, and kitchen appliances, we create a library of objects which a household robotic assistant can use while performing its tasks. Our method takes a complete 3D point cloud model as input, and computes an object model for it. The objects have states (such as open and closed), and the resultant model is accurate enough to use it in physics-based simulations, where the doors can be opened based on their hinge position. The model is built through a series of geometrical reasoning steps, namely: planar segmentation, cuboid decomposition, fixture recognition and interpretation (e. g. handles and knobs), and object classification based on object state information.

IROS Conference 2008 Conference Paper

Positioning mobile manipulators to perform constrained linear trajectories

• Franziska Zacharias
• Christoph Borst 0001
• Michael Beetz
• Gerhard Hirzinger

For mobile manipulators envisioned in home environments a kitchen scenario provides a challenging testbed for numerous skills. Diverse manipulation actions are required, e. g. simple pick and place for moving objects and constrained motions for opening doors and drawers. The robot kinematics and link limits however are restrictive. Therefore especially a constrained trajectory will not be executable from arbitrary placements of the mobile manipulator. A two stage approach is presented to position a mobile manipulator to execute constrained linear trajectories as needed for opening drawers. In a first stage, a representation of a robot arm’s reachable workspace is computed. Pattern recognition techniques are used to find regions in the workspace representation where these trajectories are possible. A set of trajectories results. In the second stage mobile manipulator placements are computed and the corresponding trajectories are checked for collisions. Compared to a brute force search through the workspace, the success rate of finding a mobile manipulator placement can be increased from 2% to 70%.

IJCAI Conference 2007 Conference Paper

• Michael Beetz
• Suat Gedikli
• Jan Bandouch
• Bernhard Kirchlechner
• Nico v. Hoyningen-Huene
• Alexander Perzylo

This paper describes ASPOGAMO, a visual tracking system that determines the coordinates and trajectories of football players in camera view based on TV broadcasts. To do so, ASPOGAMO solves a complex probabilistic estimation problem that consists of three subproblems that interact in subtle ways: the estimation of the camera direction and zoom factor, the tracking and smoothing of player routes, and the disambiguation of tracked players after occlusions. The paper concentrates on system aspects that make it suitable for operating under unconstrained conditions and in (almost) realtime. We report on results obtained in a public demonstration at RoboCup 2006 where we conducted extensive experiments with real data from live coverage of World Cup 2006 games in Germany.

ICRA Conference 2007 Conference Paper

Seamless Execution of Action Sequences

• Freek Stulp
• Wolfram Koska
• Alexis Maldonado
• Michael Beetz

One of the most notable and recognizable features of robot motion is the abrupt transitions between actions in action sequences. In contrast, humans and animals perform sequences of actions efficiently, and with seamless transitions between subsequent actions. This smoothness is not a goal in itself, but a side-effect of the evolutionary optimization of other performance measures. In this paper, we argue that such jagged motion is an inevitable consequence of the way human designers and planners reason about abstract actions. We then present subgoal refinement, a procedure that optimizes action sequences. Sub-goal refinement determines action parameters that are not relevant to why the action was selected, and optimizes these parameters with respect to expected execution performance. This performance is computed using action models, which are learned from observed experience. We integrate subgoal refinement in an existing planning system, and demonstrate how requiring optimal performance causes smooth motion in three robotic domains.

IROS Conference 2007 Conference Paper

Towards 3D object maps for autonomous household robots

• Radu Bogdan Rusu
• Nico Blodow
• Zoltán-Csaba Márton
• Alina Soos
• Michael Beetz

This paper describes a mapping system that acquires 3D object models of man-made indoor environments such as kitchens. The system segments and geometrically reconstructs cabinets with doors, tables, drawers, and shelves, objects that are important for robots retrieving and manipulating objects in these environments. The system also acquires models of objects of daily use such glasses, plates, and ingredients. The models enable the recognition of the objects in cluttered scenes and the classification of newly encountered objects. Key technical contributions include (1) a robust, accurate, and efficient algorithm for constructing complete object mod els from 3D point clouds constituting partial object views, (2) feature-based recognition procedures for cabinets, tables, and other task-relevant furniture objects, and (3) automatic inference of object instance and class signatures for objects of dally use that enable robots to reliably recognize the objects in cluttered and real task contexts. We present results from the sensor-based mapping of a real kitchen.

ICAPS Conference 2007 Conference Paper

Transformational Planning for Everyday Activity

• Armin Müller
• Alexandra Kirsch
• Michael Beetz

We propose an approach to transformational planning and learning of everyday activity. This approach is targeted at autonomous robots that are to perform complex activities such as household chore. Our approach operates on flexible and reliable plans suited for long-term activity and applies plan transformations that generate competent and high-performance robot behavior. We show as a proof of concept that general transformation rules can be formulated that achieve substantially and significantly improved performance using table setting as an example.

AAMAS Conference 2006 Conference Paper

Action Awareness - Enabling Agents to Optimize, Transform, and Coordinate Plans

• Freek Stulp
• Michael Beetz

AAMAS Conference 2006 Conference Paper

Camera-based Observation of Football Games for Analyzing Multi-agent Activities

• Michael Beetz
• Suat Gedikli
• Bernhard Kirchlechner
• Alexis Maldonado

ICRA Conference 2006 Conference Paper

Implicit Coordination in Robotic Teams using Learned Prediction Models

• Freek Stulp
• Michael Isik
• Michael Beetz

Many application tasks require the cooperation of two or more robots. Humans are good at cooperation in shared workspaces, because they anticipate and adapt to the intentions and actions of others. In contrast, multi-agent and multi-robot systems rely on communication to exchange their intentions. This causes problems in domains where perfect communication is not guaranteed, such as rescue robotics, autonomous vehicles participating in traffic, or robotic soccer. In this paper, we introduce a computational model for implicit coordination, and apply it to a typical coordination task from robotic soccer: regaining ball possession. The computational model specifies that performance prediction models are necessary for coordination, so we learn them off-line from observed experience. By taking the perspective of the team mates, these models are then used to predict utilities of others, and optimize a shared performance model for joint actions. In several experiments conducted with our robotic soccer team, we evaluate the performance of implicit coordination

ICRA Conference 2004 Conference Paper

Acquiring Models of Rectangular 3D Objects for Robot Maps

• Derik Schröter
• Michael Beetz

State-of-the-art robot mapping approaches are capable of acquiring impressively accurate 2D and 3D models of their environments. To the best of our knowledge few of them can acquire models of task-relevant objects. In this paper, we introduce a novel method for acquiring models of task-relevant objects from stereo images. The proposed algorithm applies methods from projective geometry and works for rectangular objects, which are, in office- and museum-like environments, the most commonly found subclass of geometric objects. The method is shown to work accurately and for a wide range of viewing angles and distances.

AAMAS Conference 2004 Conference Paper

RPL-Learn: Extending an Autonomous Robot Control Language to Perform Experience-based Learning

• Michael Beetz
• Alexandra Kirsch
• Armin Mueller

IROS Conference 2003 Conference Paper

Designing probabilistic state estimators for autonomous robot control

• Thorsten Schmitt
• Michael Beetz

This paper sketches and discusses design options for complex probabilistic state estimators and investigates their interactions and their impact on performance. We consider, as an example, the estimation of game states in autonomous robot soccer. We show that many factors other than the choice of algorithms determine the performance of the estimation systems. We propose empirical investigations and learning as necessary tools for the development of successful state estimation systems.

IROS Conference 2002 Conference Paper

Approximating the value function for continuous space reinforcement learning in robot control

• Sebastian Buck 0001
• Michael Beetz
• Thorsten Schmitt

Many robot learning tasks are very difficult to solve: their state spaces are high dimensional, variables and command parameters are continuously valued, and system states are only partly observable. In this paper, we propose to learn a continuous space value function for reinforcement learning using neural networks trained from data of exploration runs. The learned function is guaranteed to be a lower bound for, and reproduces the characteristic shape of, the accurate value function. We apply our approach to two robot navigation tasks, discuss how to deal with possible problems occurring in practice, and assess its performance.

IROS Conference 2002 Conference Paper

Fast image-based object localization in natural scenes

• Robert Hanek
• Thorsten Schmitt
• Sebastian Buck 0001
• Michael Beetz

In many robot applications, autonomous robots must be capable of localizing the objects they are to manipulate. In this paper we address the object localization problem by fitting a parametric curve model to the object contour in the image. The initial prior of the object pose is iteratively refined to the posterior distribution by optimizing the separation of the object and background. The local separation criteria are based on local statistics which are iteratively computed from the object and background region. No prior knowledge on color distributions is needed. Experiments show that the method is capable of localizing objects in a cluttered and textured scene even under strong variations of illumination. The method is able to localize a soccer ball within frame rate.

ICAPS Conference 2002 Conference Paper

Plan Representation for Robotic Agents

• Michael Beetz

Most robotic agents cannot fully exploit plans as resources for better problem-solving performance because of imminent limitations of their plan representations. In this paper we propose plan representations that are, for a given job, representationally and inferentially adequate and inferentially and acquisitionally efficient. We state what these properties mean in the context of robotic agents and describe how plan representations can be designed to satisfy them. The proposed plan representations have been successfully employed in several longterm experiments on autonomous robots.

AAMAS Conference 2002 Conference Paper

The AGILO Autonomous Robot Soccer Team: Computational Principles, Experiences, and Perspectives

• Michael Beetz
• Sebastian Buck
• Robert Hanek
• Thorsten Schmitt

IROS Conference 2001 Conference Paper

Cooperative probabilistic state estimation for vision-based autonomous mobile robots

• Thorsten Schmitt
• Robert Hanek
• Sebastian Buck 0001
• Michael Beetz

With the services that autonomous robots are to provide becoming more demanding, the states that the robots have to estimate become more complex. We develop and analyze a probabilistic, vision-based state estimation method for individual, autonomous robots. This method enables a team of mobile robots to estimate their joint positions in a known environment and track the positions of autonomously moving objects. The state estimators of different robots cooperate to increase the accuracy and reliability of the estimation process. This cooperation between the robots enables them to track temporarily occluded objects and to faster recover their position after they have lost track of it. The method is empirically validated based on experiments with a team of physical robots.

IROS Conference 2001 Conference Paper

Multi-robot path planning for dynamic environments: a case study

• Sebastian Buck 0001
• Ulrich Weber
• Michael Beetz
• Thorsten Schmitt

Most multi-robot navigation planning methods make assumptions about the kind of navigation problems they are to solve and the capabilities of the robots they are to control. In this paper, we propose to select problem-adequate navigation planning methods based on empirical investigations, that is, the robots should learn by experimentation to use the best planning methods. To support this development strategy we provide software tools that enable the robots to automatically learn predictive models for the performance of different navigation planning methods in a given application domain. We show, in the context of robot soccer, that the hybrid planning method which selects planning methods based on a learned predictive model outperforms the individual planning methods. The results are validated in extensive experiments using a realistic and accurate robot simulator that has learned the dynamic model of the real robots.

ECAI Conference 2000 Conference Paper

Autonomous Environment and Task Adaptation for Robotic Agents

• Michael Beetz
• Thorsten Belker

ICAPS Conference 2000 Conference Paper

Probabilistic Hybrid Action Models for Predicting Concurrent Percept-Driven Robot Behavior

• Michael Beetz
• Henrik Grosskreutz

This paper develops Probabilistic Hybrid Action Models (PHAMs), a realistic causal model for predicting the behavior generated by modern concurrent percept-driven robot plans. PHAMs represent aspects of robot behavior that cannot be represented by most action models used in AI planning: the temporal structure of continuous control processes, their non-deterministic effects, and several modes of their interferences. The main contributions of the paper are: (1) PHAMs, a model of concurrent percept-driven behavior, its formalization, and proofs that the model generates probably, qualitatively accurate predictions; and (2) a resource-efficient inference method for PHAMs based on sampling projections from probabilistic action models and state descriptions. We discuss how PHAM s can be applied to planning the course of action of an autonomous robot office courier based on analytical and experimental results.

IROS Conference 1999 Conference Paper

Controlling image processing: providing extensible, run-time configurable functionality on autonomous robots

• Tom Arbuckle
• Michael Beetz

The dynamic nature of autonomous robots' tasks requires that their image processing operations are tightly coupled to those actions within their control systems which require the visual information. While there are many image processing libraries that provide the raw image processing functionality required for autonomous robot applications, these libraries do not provide the additional functionality necessary for transparently binding image processing operations within a robot's control system. In particular such libraries lack facilities for process scheduling, sequencing, concurrent execution and resource management. The paper describes the design and implementation of an enabling extensible system-RECIPE-for providing image processing functionality in a form that is convenient for robot control together with concrete implementation examples.

ICRA Conference 1999 Conference Paper

Semi-Automatic Acquisition of Symbolically-Annotated 3D-Models of Office Environments

• Michael Beetz
• Markus Giesenschlag
• Roman Englert
• Eberhard Gülch
• Armin B. Cremers

Describes a semi-automatic method for acquiring SA3D maps, maps that contain hierarchically structured 3D models of static, task relevant objects in the environment. Map acquisition is implemented as a two step process. In the first step, the robot acquires an approximate model that represents regions that might contain objects and indicate possibly occluding objects. This approximate model is then used to compute appropriate locations from where camera images should be taken. Object models are reconstructed interactively through human operators who place wireframe model in the camera images captured by the robot. The method is implemented and validated on an autonomous mobile robot.

ICAPS Conference 1998 Conference Paper

Causal Models of Mobile Service Robot Behavior

• Michael Beetz
• Henrik Grosskreutz

Temporalprojection, the process of predicting what will happenwhena robot executesits plan, is essential for autonomous servicerobots to successfullyplan their missions. This paper describes a causal model of the behaviorexhibited by the mobilerobot RHINO whenrunningconcurrentreactive plans for performing office deliveryjobs. Themodelrepresents aspects of robot behaviorthat cannotbe representedby mostaction modelsusedin AIplanning: it representsthe temporal structure of continuouscontrolprocesses, several modesof their interferences, andvariouskindsof uncertainty. This enhancedexpressivenessenablesXFRM (McD92; BM94), a robot planningsystem, to predict, andthereforeforestall, variouskindsof behaviorflaws includingmisseddeadlineswhilst exploitingincidental opportunities. Theproposedcausal modelis experimentallyvalidatedusingthe robot andits simulator.

ECAI Conference 1998 Conference Paper

Transparent, Flexible, and Resource-adaptive Image Processing for Autonomous Service Robots

• Michael Beetz
• Tom Arbuckle
• Armin B. Cremers
• M. Mann

ICAPS Conference 1996 Conference Paper

Local Planning of Ongoing Activities

• Michael Beetz
• Drew V. McDermott

An agent that learns about the world while performing its jobs has to plan in a exible and focused manner: it has to re ect on how to perform its jobs while accomplishing them, focus on critical aspects of important subtasks, and ignore irrelevant aspects of their context. It also has to postpone planning when lacking information, reconsider its course of action when noticing opportunities, risks, or execution failures, and integrate plan revisions smoothly into its ongoing activities. In this paper, we add constructs to rpl (Reactive Plan Language) that allow for local planning of ongoing activities. The additional constructs constitute an interface between rpl and planning processes that is identical to the interface between rpl and continuous control processes like moving or grasping. The uniformity of the two interfaces and the control structures provided by rpl enable a programmer to concisely specify a wide spectrum of interactions between planning and execution.

ICAPS Conference 1994 Conference Paper

Improving Robot Plans During Their Execution

• Michael Beetz
• Drew V. McDermott

Wedescribe howour planner, XFr~f, carries out the process of anticipating and forestalling executionfallurt~s. XFI~Iis a planning systemthat is embedded in a simulated robot perfornfing a varying set of complex tasks in a cha~lging and partially unknownenvironment. XFRM revises plans controlling the robot while they are executed. Thuswheneverthe robot detects a contingency, XFRM projects the effects of the contingencyon its plan and--if necessaxy--. -revisesits plan in order to makeit more robust. Using X~’RM, the robot can performits tasks almost as efllciently as it could using efficient default plans, but muchmorerobustly. Revising default plans requires XFRM to reason about full-fledged robot plans and diagnose various kinds of plan failures that might be caused by imperfect sensing and effecting, incomplete and faulty world models, and exogenousevents. To this end, XFm~Ireasons about the structure, function, and behavior of plans, and diagnoses projected plan failures by classifying them in a taxonomyof predefined failure models. Declarative commands for goals, perceptions, a~td beliefs makethe structure of robot plans and the functions of subplans explicit and thereby provide XFRM with a (partial) modelof its plan that is used to perform hierarchical modebbaseddiagnosis.