Learning Task-Specific Generalized Convolutions in the Permutohedral Lattice

Abstract

0000−0002−7016−3820] , Martin Kiefel 2[0000−0001−9432−5428] , Peter V. Gehler 2[0000−0002−5812−4052] , Stefan Roth 1[0000−0001−9002−9832]Abstract. Dense prediction tasks typically employ encoder-decoder architectures, but the prevalent convolutions in the decoder are not imageadaptive and can lead to boundary artifacts. Different generalized convolution operations have been introduced to counteract this. We go beyond these by leveraging guidance data to redefine their inherent notion of proximity. Our proposed network layer builds on the permutohedral lattice, which performs sparse convolutions in a high-dimensional space allowing for powerful non-local operations despite small filters. Multiple features with different characteristics span this permutohedral space. In contrast to prior work, we learn these features in a task-specific manner by generalizing the basic permutohedral operations to learnt feature representations. As the resulting objective is complex, a carefully designed framework and learning procedure are introduced, yielding rich feature embeddings in practice. We demonstrate the general applicability of our approach in different joint upsampling tasks. When adding our network layer to state-of-the-art networks for optical flow and semantic segmentation, boundary artifacts are removed and the accuracy is improved.

MonoPLFlowNet: Permutohedral Lattice FlowNet for Real-Scale 3D Scene FlowEstimation with Monocular Images

“…While it is straightforward to derive v i and p i when feature positions and lattice have equal dimension, eg. 3D Lidar points input to 3D lattice [15,21,40] or 2D to 2D [1,45], it is challenging when dimensions are different, as in the case of our 2D image input to 3D lattice. The proposed Mono-PLFlowNet is designed to overcome such problem.…”

Probabilistic Pixel-Adaptive Refinement Networks

“…However, they remain restricted to a 2D grid structure. [31,56] perform image-adaptive convolutions with predefined or learned features in the high-dimensional permutohedral lattice [1]. Such convolutions are computationally expensive and thus not suitable for fast processing.…”

“…We address this issue with an advanced normalization scheme. To that end, we adapt the normalization from [56] to the context of PACs. An auxiliary array v aux = 1 with constant value 1 is defined and passed through the filtering step in Eq.…”

“…With the novel normalization, not only the number of similar pixels is taken into account but also their weighting by W. However, the normalization becomes invalid as soon as the weight W becomes negative [56]. We thus follow [56] to introduce an additional normalization weight W , which The weighting of estimates according to their pixel feature similarity K as well as their confidence c allows to remove outliers. replaces W in the normalization convolution in Eq.…”

“…skip connections [41,49] or densely connected blocks [24,32]. Additionally, different types of generalized convolutions, taking into account a highresolution RGB guidance image, have shown beneficial as part of the decoder [25,26] or in a separate upsampling and/or refinement step [31,48,56,58]. Many of the above approaches require a large number of additional parameters, are computationally expensive, or restrictive w.r.t.…”