Self-supervised Monocular Multi-robot Relative Localization with Efficient Deep Neural Networks

Abstract

Relative localization is an important ability for multiple robots to perform cooperative tasks in GPS-denied environments. This paper presents a novel autonomous positioning framework for monocular relative localization of multiple tiny flying robots. This approach does not require any groundtruth data from external systems or manual labeling. Instead, the proposed framework is able to label real-world images with 3D relative positions between robots based on another onboard relative estimation technology, using ultrawideband (UWB). After training in this self-supervised manner, the proposed deep neural network (DNN) can predict relative positions of peer robots by purely using a monocular camera. This deep learning-based visual relative localization is scalable, distributed, and autonomous. We also built an open-source and lightweight simulation pipeline by using Blender for 3D rendering, which allows synthetic image generation of other robots, and generalized training of the neural network. The proposed localization framework is tested on two real-world Crazyflie2 quadrotors by running the DNN on the onboard AIdeck (a tiny AI chip and monocular camera). All results demonstrate the effectiveness of the self-supervised multi-robot localization method. Video: https://youtu.be/7arkaIblPps

Vision-state Fusion: Improving Deep Neural Networks for Autonomous Robotics

“…In the drone-to-drone (D2D) use case, we adapt PULP-Frontnet to the different task of estimating a peer drone's pose. For the same task, Li et al [15] recently proposed also a YOLOv3based [32] CNN, but a direct comparison is not possible, since neither code or data has been made public. Nonetheless, they validate their system only on a limited test set of 48 images from a single drone, with ground-truth labels acquired through a custom UWB 3D tracking system, with unspecified accuracy.…”

“…To the best of our knowledge, SoA vision-based deep learning non-egocentric approaches do not exploit information about the robot's state within their perception process. In autonomous robotics, examples include human pose estimation [2,14], tracking of peer drones [15], or gates localization to fly through them in an autonomous drone race [16,17]. Similarly, approaches for robotic arms manipulation focus on identifying and localizing an object of interest to be grasped [1,[18][19][20].…”

“…Similarly, approaches for robotic arms manipulation focus on identifying and localizing an object of interest to be grasped [1,[18][19][20]. Ultimately, both robotic domains solve their deep learning-based perception task by employing only visual feeds, e.g., front-looking monocular-camera [2,[14][15][16][17], eye-to-hand (i.e., fixed in space) [1,18,19] or eyein-hand (i.e., attached to the end effector) [20] cameras.…”

Self-Supervised Learning of Visual Robot Localization Using LED State Prediction as a Pretext Task

“…T HE ability to estimate the position of a target robot in a video feed is crucial for many robotics tasks [1], [2], [3]. State-of-the-art (SoA) approaches use deep learning techniques based on Convolutional Neural Networks (CNNs) [4]: given a camera frame, they segment the target robot, regress the coordinates of its bounding box or its position in the image. Training these approaches to handle new robots or environments requires extensive labeled datasets, which are time-consuming and expensive to acquire, often relying on specialized hardware, e.g., motion tracking systems, to generate ground truth labels.…”

Efficient 2D Graph SLAM for Sparse Sensing

“…Virtually all state-of-the-art 2D SLAM solutions today are designed for robots with dense and accurate sensors such as laser range-finders (LiDARs). On the contrary, recent work has shown that small, agile, and cheap nano drones demonstrate potential to carry out dangerous indoor exploration missions [2] [3]. These nano drones have limited battery and carrying capacity, and it is only possible to mount low-power sensors that can only provide sparse and noisy measurements.…”