Learning by Analogy: Reliable Supervision From Transformations for Unsupervised Optical Flow Estimation

Liu, Liang; Zhang, Jiangning; He, Ruifei; Liu, Yong; Wang, Yabiao; Tai, Ying; Luo, Donghao; Wang, Chengjie; Li, Jilin; Huang, Feiyue

doi:10.1109/cvpr42600.2020.00652

Cited by 170 publications

(158 citation statements)

References 31 publications

Supporting

Mentioning

157

Contrasting

Order By: Relevance

“…Each optical flow was extracted using the Farneback method, except for DTV-Net. For DTV-Net, we use optical flow extracted by ARFLow [23], following the original method [22].…”

Section: ) Optical Flow Evaluationmentioning

confidence: 99%

Animating Cloud Images With Flow Style Transfer

Kurisaki

Kawamoto

2021

IEEE Access

View full text Add to dashboard Cite

We propose a method for animating static images using a generative adversarial network (GAN). Given a source image depicting a cloud image and a driving video sequence depicting a moving cloud image, our framework generates a video in which the source image is animated according to the driving sequence. By inputting the source image and optical flow of the driving video into the generator, a video is generated that is conditioned by the optical flow. The optical flow enables the application of the captured motion of clouds in the source image. Further, we experimentally show that the proposed method is more effective than the existing methods for animating a keypoint-less video (in which the keypoints cannot be explicitly determined) such as a moving cloud image. Furthermore, we show an improvement in the quality of the generated video due to the use of optical flow in the video reconstruction.INDEX TERMS Image animation, video generation, generative adversarial networks, optical flow.

show abstract

“…Each optical flow was extracted using the Farneback method, except for DTV-Net. For DTV-Net, we use optical flow extracted by ARFLow [23], following the original method [22].…”

Section: ) Optical Flow Evaluationmentioning

confidence: 99%

Animating Cloud Images With Flow Style Transfer

Kurisaki

Kawamoto

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Recent unsupervised optical flow estimation approaches have attracted much attention, because their advantage in not requiring ground truth enables them to be easily deployed in real-world applications [8,9,10,11,12]. However, their performance in challenging regions, such as partially occluded or texture-less regions, is often unsatisfactory [10,13]. The underlying cause of this performance degradation is threefold: 1) The popular coarse-to-fine framework [12,13] is often sensitive to noises in the flow initialization from the preceding pyramid level, and the challenging regions can introduce errors in the flow estimations, which in turn propagate to subsequent levels.…”

Section: Introductionmentioning

confidence: 99%

“…lenging regions for unsupervised optical flow estimation, such as occlusion reasoning [9,10] and self-supervision [11,12,13]. These strategies generally train a single network to provide prior information.…”

Section: Introductionmentioning

confidence: 99%

“…† indicates the network using more than two frames. The best result is shown in bold font.`````````S trategy Network 1)UnFlow-Net [9] 2)DDFlow-Net [11] 3)SelFlow-Net † [12] 4)ARFlow-Net [13] 5)AMFlow (Ours) a) UnFlowStrat [9] 8 other training strategies, our co-teaching strategy achieves the best performance when employed in the same network architecture, i.e., our AMFlow, which further demonstrates the superiority of our co-teaching strategy over other strategies for unsupervised training.…”

mentioning

confidence: 99%

“…Examples of the MPI Sintel Clean, KITTI Flow 2012 and KITTI Flow 2015 benchmarks, where rows (a) and (b) on columns (1)-(3) show the flow estimations and the corresponding error maps of (1) ARFlow-mv[13], (2) SelFlow[12] and (3) our CoT-AMFlow, respectively. Significantly improved regions are highlighted with green dashed boxes.…”

mentioning

confidence: 99%

See 2 more Smart Citations

CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation

Wang¹,

Fan²,

Liu³

2020

Preprint

View full text Add to dashboard Cite

The interpretation of ego motion and scene change is a fundamental task for mobile robots. Optical flow information can be employed to estimate motion in the surroundings. Recently, unsupervised optical flow estimation has become a research hotspot. However, unsupervised approaches are often easy to be unreliable on partially occluded or texture-less regions. To deal with this problem, we propose CoT-AMFlow in this paper, an unsupervised optical flow estimation approach. In terms of the network architecture, we develop an adaptive modulation network that employs two novel module types, flow modulation modules (FMMs) and cost volume modulation modules (CMMs), to remove outliers in challenging regions. As for the training paradigm, we adopt a co-teaching strategy, where two networks simultaneously teach each other about challenging regions to further improve accuracy. Experimental results on the MPI Sintel, KITTI Flow and Middlebury Flow benchmarks demonstrate that our CoT-AMFlow outperforms all other state-of-the-art unsupervised approaches, while still running in real time. Our project page is available at https://sites.google.com/view/cot-amflow.

show abstract

DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Zhang

Liu

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper presents a novel end-to-end dynamic time-lapse video generation framework, named DTVNet, to generate diversified time-lapse videos from a single landscape image, which are conditioned on normalized motion vectors. The proposed DTVNet consists of two submodules: Optical Flow Encoder (OFE) and Dynamic Video Generator (DVG). The OFE maps a sequence of optical flow maps to a normalized motion vector that encodes the motion information inside the generated video. The DVG contains motion and content streams that learn from the motion vector and the single image respectively, as well as an encoder and a decoder to learn shared content features and construct video frames with corresponding motion respectively. Specifically, the motion stream introduces multiple adaptive instance normalization (AdaIN) layers to integrate multi-level motion information that are processed by linear layers. In the testing stage, videos with the same content but various motion information can be generated by different normalized motion vectors based on only one input image. We further conduct experiments on Sky Time-lapse dataset, and the results demonstrate the superiority of our approach over the state-of-the-art methods for generating high-quality and dynamic videos, as well as the variety for generating videos with various motion information. 1

show abstract

Learning by Analogy: Reliable Supervision From Transformations for Unsupervised Optical Flow Estimation

Cited by 170 publications

References 31 publications

Animating Cloud Images With Flow Style Transfer

Animating Cloud Images With Flow Style Transfer

CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation

DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Contact Info

Product

Resources

About