Persistent self-supervised learning: From stereo to monocular vision for obstacle avoidance

Abstract

Self-Supervised Learning (SSL) is a reliable learning mechanism in which a robot uses an original, trusted sensor cue for training to recognize an additional, complementary sensor cue. We study for the first time in SSL how a robot's learning behavior should be organized, so that the robot can keep performing its task in the case that the original cue becomes unavailable. We study this persistent form of SSL in the context of a flying robot that has to avoid obstacles based on distance estimates from the visual cue of stereo vision. Over time it will learn to also estimate distances based on monocular appearance cues. A strategy is introduced that has the robot switch from stereo vision based flight to monocular flight, with stereo vision purely used as 'training wheels' to avoid imminent collisions. This strategy is shown to be an effective approach to the 'feedbackinduced data bias' problem as also experienced in learning from demonstration. Both simulations and real-world experiments with a stereo vision equipped AR drone 2.0 show the feasibility of this approach, with the robot successfully using monocular vision to avoid obstacles in a 5 × 5m room. The experiments show the potential of persistent SSL as a robust learning approach to enhance the capabilities of robots. Moreover, the abundant training data coming from the own sensors allows to gather large data sets necessary for deep learning approaches.

Artificial Neural Network-Based Flight Control Using Distributed Sensors on Fixed-Wing Unmanned Aerial Vehicles

“…For this control approach to be successfully used in a real-world environment, a strategy that takes advantage of the behavior learned under controlled conditions needs to be developed. A possible solution is to use persistent self-supervised learning [32], where the system uses previously learned input-output mappings to recognize additional, complementary information.…”

A Survey on Swarming With Micro Air Vehicles: Fundamental Challenges and Constraints

“…Self-supervised learning of distance from monocular images can also be accomplished without the need to crash, but with the aid of an additional sensor. Lamers et al ( 2016 ) did this by exploiting an infrared range sensor, and van Hecke et al ( 2018 ) applied this to see distances with one single camera by learning a behavior that used a stereo-camera. This is useful if the stereo-camera were to malfunction and suddenly become monocular.…”

Fusion of Stereo and Still Monocular Depth Estimates in a Self-Supervised Learning Context

“…Self-supervised learning of monocular depth estimation is a more recent topic [27], [25], [16]. [25] conducted the first study where stereo vision was used as supervisory input to teach a single mono estimator how to predict depths.…”

“…Self-supervised learning of monocular depth estimation is a more recent topic [27], [25], [16]. [25] conducted the first study where stereo vision was used as supervisory input to teach a single mono estimator how to predict depths. However, as the focus of the study was more on the behavioral aspects of SSL and all algorithms had to run on a computationally limited robot in space [26], only the average depth of the scene was learned.…”