Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations

Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https: //github. com/ease-crc/vr-program-synthesis.

Authors

• Benjamin Alt ArtiMinds Robotics; University of Bremen
• Franklin Kenghagho Kenfack University of Bremen
• Andrei Haidu University of Bremen
• Darko Katic ArtiMinds Robotics
• Rainer Jäkel ArtiMinds Robotics
• Michael Beetz Universität Bremen

Keywords

• Applications of KR in robotics
• Cognitive robot architectures
• Development, deployment, and evaluation of KR systems to solve real-world problems
• KR and cognitive robotics