Monocular Depth Estimation via Listwise Ranking using the Plackett-Luce Model

Abstract

In many real-world applications, the relative depth of objects in an image is crucial for scene understanding, e.g., to calculate occlusions in augmented reality scenes. Predicting depth in monocular images has recently been tackled using machine learning methods, mainly by treating the problem as a regression task. Yet, being interested in an order relation in the first place, ranking methods suggest themselves as a natural alternative to regression, and indeed, ranking approaches leveraging pairwise comparisons as training information ("object A is closer to the camera than B") have shown promising performance on this problem. In this paper, we elaborate on the use of so-called listwise ranking as a generalization of the pairwise approach. Listwise ranking goes beyond pairwise comparisons between objects and considers rankings of arbitrary length as training information. Our approach is based on the Plackett-Luce model, a probability distribution on rankings, which we combine with a state-of-the-art neural network architecture and a sampling strategy to reduce training complexity. An empirical evaluation on benchmark data in a "zero-shot" setting demonstrates the effectiveness of our proposal compared to existing ranking and regression methods.

Single Image Depth Estimation: An Overview

“…An important theme we see here is: Classification and ordinality A number of works decided to formulate the depth estimation problem in a way that does not require the system to estimate the exact depth value [2], [12], [20], [27], [40], [41], [42], [43], [44], [45]. While estimating the relative depth instead of the absolute depth is an alternative, estimating the depth range in a classification setting can be done with success.…”

“…There are two important issues to consider for depth in the wild problem. The first issue a system faces that aims to work in [20] softmax-loss -custom Chen et al [12] pairwise ranking -Hourglass Xian et al [40] pairwise ranking random + hard pairs EncDecResNet Chen et al [41] pairwise ranking random + hard pairs EncDecResNet Xian et al [44] pairwise ranking random + instance edges + image edges EncDecResNet Mertan et al [42] listwise ranking random EncDecResNet Lienen et al [45] listwise ranking random EfficientNet variant Mertan et al [43] distributional ranking random EncDecResNet the wild is that the depth ranges may vary quite a lot which is a problem for the learning process. While the range of absolute depth values for an image of a table may be between 10 cm and a couple of meters, the image of a touristic place such as "Eiffel Tower" contains pixels that are a hundred meters away, if they have a depth value at all (sky regions can be considered infinitely far away).…”

“…Instead of improving the sampling strategy, Mertan et al [42] and Lienen et al [45] try to improve the loss function. Both work implements the same idea.…”