UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss

Meister, Simon; Hur, Junhwa; Roth, Stefan

doi:10.1609/aaai.v32i1.12276

Cited by 408 publications

(94 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…summa-rizes the quantitative results. We include comparisons with several baseline methods for unsupervised optical flow estimation: Unsup (Jason, Harley, and Derpanis 2016) and Un-Flow (Meister, Hur, and Roth 2018) and its variants, which are also trained on V1FlyingObjects. The proposed model achieves better performance compared to baseline methods.…”

Section: Unsupervised Learningmentioning

confidence: 99%

Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion

Gao

Xie

Huang

et al. 2022

AAAI

View full text Add to dashboard Cite

This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1). The model couples the following two components: (1) the vector representations of local contents of images and (2) the matrix representations of local pixel displacements caused by the relative motions between the agent and the objects in the 3D scene. When the image frame undergoes changes due to local pixel displacements, the vectors are multiplied by the matrices that represent the local displacements. Thus the vector representation is equivariant as it varies according to the local displacements. Our experiments show that our model can learn Gabor-like filter pairs of quadrature phases. The profiles of the learned filters match those of simple cells in Macaque V1. Moreover, we demonstrate that the model can learn to infer local motions in either a supervised or unsupervised manner. With such a simple model, we achieve competitive results on optical flow estimation.

show abstract

Section: Unsupervised Learningmentioning

confidence: 99%

Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion

Gao

Xie

Huang

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…The unsupervised methods rely on the brightness constancy constraint and establish a loss function on the original image pixels. In addition to the basic brightness constancy loss, there are many other techniques used for unsupervised optical flow learning, such as smooth constraint (Ren et al 2017;Jason, Harley, and Derpanis 2016), census transformation (Meister, Hur, and Roth 2018). One difficulty of the unsupervised methods is that the pixels in the occluded area do not satisfy the brightness constancy assumption.…”

Section: Optical Flowmentioning

confidence: 99%

Promoting Single-Modal Optical Flow Network for Diverse Cross-Modal Flow Estimation

Zhou

Tan

Yan

2022

AAAI

View full text Add to dashboard Cite

In recent years, optical flow methods develop rapidly, achieving unprecedented high performance. Most of the methods only consider single-modal optical flow under the well-known brightness-constancy assumption. However, in many application systems, images of different modalities need to be aligned, which demands to estimate cross-modal flow between the cross-modal image pairs. A lot of cross-modal matching methods are designed for some specific cross-modal scenarios. We argue that the prior knowledge of the advanced optical flow models can be transferred to the cross-modal flow estimation, which may be a simple but unified solution for diverse cross-modal matching tasks. To verify our hypothesis, we design a self-supervised framework to promote the single-modal optical flow networks for diverse corss-modal flow estimation. Moreover, we add a Cross-Modal-Adapter block as a plugin to the state-of-the-art optical flow model RAFT for better performance in cross-modal scenarios. Our proposed Modality Promotion Framework and Cross-Modal Adapter have multiple advantages compared to the existing methods. The experiments demonstrate that our method is effective on multiple datasets of different cross-modal scenarios.

show abstract

“…Early networks are trained on the synthetic dataset [13], that is because the ground truth of optical flow in the real world is difficult to obtain, which increases the difficulty for supervision and estimation. As a result, the method of unsupervised learning on real-world data has attracted much attention [20], [41], [21], [22], [23], [24].…”

Section: B Unsupervised Estimation Of Optical Flowmentioning

confidence: 99%

“…Most supervised research on optical flow has been done on synthetic datasets [18], as well as scene flow [16], [17]. That is why unsupervised learning is as important to scene flow as optical flow [19], [20], [21], [22], [23], [24]. The realworld image data can be used in the unsupervised training of scene flow with suitable losses as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Learning of 3D Scene Flow from Monocular Camera

Wang

Tian

Wang

2021

2021 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Scene flow represents the motion of points in the 3D space, which is the counterpart of the optical flow that represents the motion of pixels in the 2D image. However, it is difficult to obtain the ground truth of scene flow in the real scenes, and recent studies are based on synthetic data for training. Therefore, how to train a scene flow network with unsupervised methods based on real-world data shows crucial significance. A novel unsupervised learning method for scene flow is proposed in this paper, which utilizes the images of two consecutive frames taken by monocular camera without the ground truth of scene flow for training. Our method realizes the goal that training scene flow network with real-world data, which bridges the gap between training data and test data and broadens the scope of available data for training. Unsupervised learning of scene flow in this paper mainly consists of two parts: (i) depth estimation and camera pose estimation, and (ii) scene flow estimation based on four different loss functions. Depth estimation and camera pose estimation obtain the depth maps and camera pose between two consecutive frames, which provide further information for the next scene flow estimation. After that, we used depth consistency loss, dynamic-static consistency loss, Chamfer loss, and Laplacian regularization loss to carry out unsupervised training of the scene flow network. To our knowledge, this is the first paper that realizes the unsupervised learning of 3D scene flow from monocular camera. The experiment results on KITTI show that our method for unsupervised learning of scene flow meets great performance compared to traditional methods Iterative Closest Point (ICP) and Fast Global Registration (FGR). The source code is available at: https://github.com/IRMVLab/3DUnMonoFlow.

show abstract

UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss

Cited by 408 publications

References 32 publications

Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion

Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion

Promoting Single-Modal Optical Flow Network for Diverse Cross-Modal Flow Estimation

Unsupervised Learning of 3D Scene Flow from Monocular Camera

Contact Info

Product

Resources

About