SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings

Abstract

The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene understanding and visual odometry, which are key components in autonomous and robotic systems. Approaches based on the state of the art vision transformer architectures are extremely deep and complex not suitable for realtime inference operations on edge and autonomous systems equipped with low resources (i.e. robot indoor navigation and surveillance). This paper presents SPEED, a Separable Pyramidal pooling EncodEr-Decoder architecture designed to achieve real-time frequency performances on multiple hardware platforms. The proposed model is a fast-throughput deep architecture for MDE able to obtain depth estimations with high accuracy from low resolution images using minimum hardware resources (i.e. edge devices). Our encoderdecoder model exploits two depthwise separable pyramidal pooling layers, which allow to increase the inference frequency while reducing the overall computational complexity. The proposed method performs better than other fast-throughput architectures in terms of both accuracy and frame rates, achieving realtime performances over cloud CPU, TPU and the NVIDIA Jetson TX1 on two indoor benchmarks: the NYU Depth v2 and the DIML Kinect v2 datasets.INDEX TERMS Computer vision, monocular depth estimation, fast-throughput, edge devices

Lightweight and Energy-Aware Monocular Depth Estimation Models for IoT Embedded Devices: Challenges and Performances in Terrestrial and Underwater Scenarios

“…We can observe this fact in Figure 3 (third column), where missing depth measurements are represented as yellow pixels. As can be seen from the reported results, despite the high estimation error achieved by all the methods with respect to the terrestrial dataset, the proposed MobileNetV3 with only of training parameters outperforms both [ 12 ] and [ 5 ] while obtaining the same RMSE and a higher to [ 11 ]. In addition, the proposed model achieves a boost on the inference frequency equal to with respect to [ 11 ] on the same benchmark hardware.…”

“…We designed those models in compliance with two essential concepts: the speed to maximize the inference frequency on embedded devices, and the robustness to maximize the estimation accuracy across two datasets [ 14 , 26 ]. Our solution exploits an encoder–decoder model, similar to previous related works, such as [ 11 , 12 ]. In more detail, we perform an in-depth study on lightweight encoders pre-trained on ImageNet [ 27 ] to improve their generalization capabilities.…”

“…Papa et al. [ 11 ] propose SPEED, a separable pyramidal pooling architecture designed to achieve real-time frequency performances on both cloud CPU, TPU, and edge devices, e.g., NVIDIA-TX1 GPU. The model makes extensive use of depthwise separable convolutions to achieve fast and accurate depth estimations.…”

“…Regarding monocular depth estimation, only a few works propose a solution for porting such complex tasks on low-resource platforms. There are two main approaches: [ 8 , 9 , 10 ] that focus on MDE on microcontroller and ARM-powered devices without taking into account the inference frequency and [ 11 , 12 , 13 ], which analyze the inference performances of MDE on low-power embedded GPUs. Furthermore, MDE methods are usually trained in supervised learning strategies on indoor and outdoor terrestrial datasets such as [ 14 , 15 ].…”

A Review of Embedded Machine Learning Based on Hardware, Application, and Sensing Scheme

“…It involves the use of pixel shape and orientation for the identification of the distance of objects within 2D images and video from the device that recorded it. Its utility is mainly in photography and depth estimation for self-driving vehicles, while within our sources, it was mostly used for personal projects such as in [ 88 ]. The performance of these applications is covered in Table 4 as well as a comparison graph being provided in Figure 4 .…”