TerraPN: Unstructured Terrain Navigation using Online Self-Supervised Learning

Abstract

We present CoNVOI, a novel method for autonomous robot navigation in real-world indoor and outdoor environments using Vision Language Models (VLMs). We employ VLMs in two ways: first, we leverage their zeroshot image classification capability to identify the context or scenario (e.g., indoor corridor, outdoor terrain, crosswalk, etc) of the robot's surroundings, and formulate context-based navigation behaviors as simple text prompts (e.g. "stay on the pavement"). Second, we utilize their state-of-the-art semantic understanding and logical reasoning capabilities to compute a suitable trajectory given the identified context. To this end, we propose a novel multi-modal visual marking approach to annotate the obstacle-free regions in the RGB image used as input to the VLM with numbers, by correlating it with a local occupancy map of the environment. The marked numbers ground image locations in the real-world, direct the VLM's attention solely to navigable locations, and elucidate the spatial relationships between them and terrains depicted in the image to the VLM. Next, we query the VLM to select numbers on the marked image that satisfy the context-based behavior text prompt, and construct a reference path using the selected numbers. Finally, we propose a method to extrapolate the reference trajectory when the robot's environmental context has not changed to prevent unnecessary VLM queries. We use the reference trajectory to guide a motion planner, and demonstrate that it leads to human-like behaviors (e.g. not cutting through a group of people, using crosswalks, etc.) in various real-world indoor and outdoor scenarios. We perform several ablations and navigation comparisons and demonstrate that CoNVOI's trajectories are most similar to human teleoperated ground truth in terms of Fréchet distance (9.7-58.2% closer), lowest path errors (up to 88.13% lower), and up to 86.09% lower % of unacceptable paths.

Challenges and Solutions for Autonomous Ground Robot Scene Understanding and Navigation in Unstructured Outdoor Environments: A Review

“…Several other research works [142][143][144][145] have used machine learning to achieve robustness in path following. In recent years, many Reinforcement Learning (RL) and imitation learning methods and approaches based on self-supervised learning [146][147][148][149] have been applied in mobile robot navigation policy design and for training supervision. Many classical modular and deep learning-based approaches have been utilised in the navigation modules of outdoor mobile robots [150].…”

Terrain-aware autonomous robot navigation in outdoor environments

“…Our terrain surface perception branch incorporates a self-supervised learning strategy proposed in our previous work TerraPN 11 to quantify (as a surface cost map C s t ) the surface properties (traction, bumpiness, deformability, etc.) of outdoor terrains directly from robot-terrain interactions.…”

“…9,10 In addition to the terrain elevation, its surface properties such as texture, bumpiness, and deformability affect the smoothness of the navigation task. 11 For instance, a surface's texture affects the traction experienced by the robot, its bumpiness determines the vibrations experienced, and granularity/deformability determines whether a robot could get stuck or experience wheel slips (e.g. in sand or mud).…”

“…12,13 Hence, navigating on rough/non-smooth terrains could lead to higher vibrations, 14 higher energy consumption, 15 odometry errors (i.e., due to wheel slips), and motion blur in RGB images which could significantly affect the navigation performance. 16,17 Hence, understanding terrain surface properties (i.e., surface awareness) and performing navigation along smoother regions is equally important as elevation awareness during outdoor navigation tasks.…”

Real-Time Drift-Driving Control for an Autonomous Vehicle: Learning from Nonlinear Model Predictive Control via a Deep Neural Network

“…They primarily used human supervision to assess the difficulty of the terrain to be traversed and to provide the vehicle with online maps of traversable regions. More recently, researchers have focused on self-supervised approaches that aim to learn traversability directly from real driving experience [16], [17]. Unfortunately, all these methods may suffer from information loss when dealing with challenging terrain, because they tend to simplify traversability as a scalar value or a categorical label.…”