droidlet: modular, heterogenous, multi-modal agents

In recent years, there have been significant advances in building end-to-end Machine Learning (ML) systems that learn at scale. But most of these systems are: (a) isolated (perception, speech, or language only); (b) trained on static datasets. On the other hand, in the field of robotics, large-scale learning has always been difficult. Supervision is hard to gather and real world physical interactions are expensive. In this work we introduce and open-source droidlet, a modular, heterogeneous agent architecture and platform. It allows us to exploit both large-scale static datasets in perception and language and sophisticated heuristics often used in robotics; and provides tools for interactive annotation. Furthermore, it brings together perception, language and action onto one platform, providing a path towards agents that learn from the richness of real world interactions.

Authors

• Anurag Pratik
• Soumith Chintala
• Kavya Srinet
• Dhiraj Gandhi
• Rebecca Qian
• Yuxuan Sun 0004
• Ryan Drew
• Sara Elkafrawy
• Anoushka Tiwari
• Tucker Hart
• Mary Williamson
• Abhinav Gupta 0001
• Arthur Szlam

Keywords

• Automation
• Annotations
• Conferences
• Buildings
• Human-robot interaction
• Machine learning
• Tools
• Dataset Statistics
• Real Interactions
• Machine Learning Models
• Modularity
• Long-term Goals
• Lifelong Learning
• Physical World
• Interactive Learning
• Memory System
• Pose Estimation
• Self-supervised Learning
• Modular Design
• Instance Segmentation
• Robotic Platform
• Face Detection
• Low-level Control
• Laser Pointer
• Machine Learning Pipeline
• Domain-specific Languages
• Task Queue
• Perception Module
• Ecosystem Research
• Reference Object
• Model Of Perception
• Face Recognition
• Logic Model
• Object Detection
• Proprioceptive
• Perceptual System