droidlet: modular, heterogenous, multi-modal agents

Abstract

In recent years, there have been significant advances in building end-to-end Machine Learning (ML) systems that learn at scale. But most of these systems are: (a) isolated (perception, speech, or language only); (b) trained on static datasets. On the other hand, in the field of robotics, large-scale learning has always been difficult. Supervision is hard to gather and real world physical interactions are expensive.In this work we introduce and open-source droidlet, a modular, heterogeneous agent architecture and platform. It allows us to exploit both large-scale static datasets in perception and language and sophisticated heuristics often used in robotics; and provides tools for interactive annotation. Furthermore, it brings together perception, language and action onto one platform, providing a path towards agents that learn from the richness of real world interactions.

A Data Source for Reasoning Embodied Agents

“…With some probability, the agent executes a command (to either: build an object, destroy an object, move to a location, dig a hole, follow an NPC, etc.) The execution of a command is scripted; the task executor is from the Minecraft agent in (Pratik et al 2021). Whether or not the agent executes a task, the world steps a fixed number of times (so, e.g., NPCs may move or act).…”

“…Following (Pratik et al 2021), the environment is presented to the agent as an object-centered key-value store. Each object, NPC, and the agent's self have a "memid" keying a data structure that depends on the object type, and may contain string data (for example a name) or float or integer data (e.g.…”

“…We build our data generator on top of the Droidlet agent (Pratik et al 2021), in part using the grammar in (Srinet et al 2020) to generate the queries and using the Droidlet agent memory to execute them. This work points the way towards using neural networks to execute the functions of the Droidlet agent memory (and hopefully can be used as a resource for training other agent-memory models).…”