Navigating to objects in the real world

Abstract

Semantic navigation is necessary to deploy mobile robots in uncontrolled environments such as homes or hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, whereas modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. However, learned visual navigation policies have predominantly been evaluated in sim, with little known about what works on a robot. We present a large-scale empirical study of semantic visual navigation methods comparing representative methods with classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We found that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% sim to a 23% real-world success rate because of a large image domain gap between sim and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: Modularity and abstraction in policy design enable sim-to-real transfer. For researchers, we identify two key issues that prevent today’s simulators from being reliable evaluation benchmarks—a large sim-to-real gap in images and a disconnect between sim and real-world error modes—and propose concrete steps forward.

Modular-integrative modeling: a new framework for building brain models that blend biological realism and functional performance

“…Importantly, modules allow the combination of differentiated learning paradigms, such as a basal ganglia model with reinforcement learning connected to a cerebellar model with supervised learning. Implementation-wise, as in modular deep learning, components can thus be optimized while retaining their overall context to ensure that solutions meet global task requirements as well as constraints imposed by other components [ 12 ]. This is highly efficient and generalizes better than end-to-end optimization of the full model [ 13 ].…”

Temporally extended successor feature neural episodic control

“…RL has garnered significant interest in both the research and industrial sectors, particularly when combined with deep learning methods in a framework known as deep reinforcement learning 5 , 5 . It finds applications in various fields related to human intelligence, such as gaming 5 – 12 , autonomous driving 13 , 14 , robotics 15 , 16 , software testing 17 , 18 , quantum control 19 – 21 , and more. However, existing deep reinforcement learning algorithms often require large amounts of data and millions of training steps to achieve human-level performance, which is considered a challenge due to their low sample efficiency.…”

Decomposing user-defined tasks in a reinforcement learning setup using TextWorld

“…The overall goal of this field of AI is to deliver robotic agents that carry out tasks, manipulate objects, and can serve people in their daily routines. There are multitudes of challenges arising from this field, such as navigation (Gervet et al, 2022), natural language processing (NLP) (Shridhar et al, 2020a), simulation (Weihs et al, 2020), computer vision (Anderson et al, 2018), motion and task planning (Garrett et al, 2021), manipulation challenges (Xie et al, 2019), challenges involving all of these (Bohren et al, 2011), and even hardware limitations, just to name a few. However, another perhaps often undervalued challenge is breaking down the inherently complex language commands into simpler, more manageable, sub-commands or "sub-tasks. "…”