Rebalancing gradient to improve self-supervised co-training of depth, odometry and optical flow predictions

Abstract

We present CoopNet, an approach that improves the cooperation of co-trained networks by dynamically adapting the apportionment of gradient, to ensure equitable learning progress. It is applied to motion-aware self-supervised prediction of depth maps, by introducing a new hybrid loss, based on a distribution model of photo-metric reconstruction errors made by, on the one hand the depth + odometry paired networks, and on the other hand the optical flow network. This model essentially assumes that the pixels from moving objects (that must be discarded for training depth and odometry), correspond to those where the two reconstructions strongly disagree. We justify this model by theoretical considerations and experimental evidences. A comparative evaluation on KITTI and CityScapes datasets shows that CoopNet improves or is comparable to the state-of-the-art in depth, odometry and optical flow predictions. Our code is available here: https://github.com/mhariat/CoopNet.

GloFP-MSF: Monocular Scene Flow Estimation with Global Feature Perception

“…By utilizing advanced algorithms and techniques, scene flow estimation allows us to reconstruct and map the motion of objects across a sequence of images or frames, which has broad applications in various fields, including autonomous driving [1], action recognition [2], and virtual reality [3]. Many scene flow estimation methods based on various types of input data have recently been proposed, such as image sequence [4] [5], 3D point clouds [6] [7]. However, the acquisition of 3D point cloud data usually requires expensive sensor equipment, so the research in this paper focuses on the scene flow estimation methods whose input is image sequence.…”