Indoor Relocalization in Challenging Environments With Dual-Stream Convolutional Neural Networks

Abstract

This paper presents an indoor relocalization system using a dual-stream Convolutional Neural Network (CNN) with both color images and depth images as the network inputs. Aiming at the pose regression problem, a deep neural network architecture for RGB-D images is introduced, a training method by stages for the dual-stream CNN is presented, different depth image encoding methods are discussed and a novel encoding method is proposed. By introducing the range information into the network through a dual-stream architecture, we not only improved the relocalization accuracy by about 20% compared with the state-of-the-art deep learning method for pose regression, but also greatly enhance the system robustness in challenging scenes such as large scale, dynamic, fast movement and nighttime environments. To the best of our knowledge, this is the first work to solve the indoor relocalization problems based on deep CNNs with RGB-D camera. The method is first evaluated on the Microsoft 7-Scenes dataset to show its advantage in accuracy compared with other CNNs. Large scale indoor relocalization is further presented using our method. The experimental results show that 0.3m in position and 4 • in orientation accuracy could be obtained. Finally, this method is evaluated on challenging indoor datasets collected from motion capture system. The results show that the relocalization performance is hardly affected by dynamic objects, motion blur or night-time environments. Note to Practitioners-This work was motivated by the limitations of the existing indoor relocalization technology that is significant for mobile robot navigation. By using this technology, robots can infer where they are in a previously visited place. Previous visual localization methods can hardly be put into wide application for the reason that they have strict requirements for the environments. When faced with challenging scenes such as large scale environments, dynamic objects, motion blur caused by fast movement, night-time environments or other appearance changed scenes, most existing methods tend to fail. This paper introduces deep learning into the indoor relocalization problem, uses dual-stream CNN (depth stream and color stream) to realize 6-DOF pose regression in an end-to-end manner. The localization error is about 0.3m and 4 • in a large scale indoor environments. And what is more important, the proposed system does not lose efficiency in some challenging scenes. The proposed encoding method of depth images can also be adopted in other Deep Neural Networks with RGB-D cameras as the sensor.

UAS Navigation with SqueezePoseNet—Accuracy Boosting for Pose Regression by Data Augmentation

“…Furthermore, to improve a potential navigation approach, one may support the input of a CNN with additional data. Combining RGB data and depth data in a dual stream CNN showed further improvements of the localization results [29].…”

Squeezeposenet: Image Based Pose Regression With Small Convolutional Neural Networks for Real Time Uas Navigation

“…Furthermore, to improve a potential navigation approach, one may support the CNNs input with additional data. Combining RGB data and depth data in a dual stream CNN showed further improvements of the localization results (Li et al, 2017).…”

Application of Augmented Reality and Robotic Technology in Broadcasting: A Survey

“…Since dense point-based approach highly relies on powerful hardware, some researchers have introduced semi-dense algorithms, e.g., Semi-dense Visual Odometry has been utilized in mobile AR applications [38]. Generally, consumer-level RGB-D cameras are sensitive to noise and not very accurate, and frequently used for small AR workspace and academic research [39]. For the large scale of AR applications, it is necessary to use multiple RGB-D sensors.…”

“…For the large scale of AR applications, it is necessary to use multiple RGB-D sensors. For example, the "RoomAlive" from Generally, consumer-level RGB-D cameras are sensitive to noise and not very accurate, and frequently used for small AR workspace and academic research [39]. For the large scale of AR applications, it is necessary to use multiple RGB-D sensors.…”