PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

Sun, Deqing; Yang, Xiaodong; Li, Mingyu; Kautz, Jan

doi:10.1109/cvpr.2018.00931

Cited by 2,463 publications

(2,209 citation statements)

References 57 publications

Supporting

Mentioning

2,195

Contrasting

Unclassified

Order By: Relevance

“…Learning from synthetic data. Training the model on large-scale synthetic datasets has been extensively studied in semantic segmentation [44,45,17,16,9,19,37,38,56], multi-view stereo [20], depth estimation [51], optical flow [43,21,23], amodal segmentation [18], and object detection [9,33]. In our work, we show that the proposed cross-domain consistency loss can be applied not only to synthetic-to-real adaptation but to real-to-real adaptation tasks as well.…”

Section: Related Workmentioning

confidence: 99%

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

Chen

Lin

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

312

199

View full text Add to dashboard Cite

Semantic seg. Depth prediction Optical flow Labeled examples (source domain) Input (target domain) Output Figure 1: Applications of the proposed method. Our method has the applications ranging from semantic segmentation (top row), depth prediction (middle row), to optical flow estimation (bottom row). AbstractUnsupervised domain adaptation algorithms aim to transfer the knowledge learned from one domain to another (e.g., synthetic to real images). The adapted representations often do not capture pixel-level domain shifts that are crucial for dense prediction tasks (e.g., semantic segmentation). In this paper, we present a novel pixel-wise adversarial domain adaptation algorithm. By leveraging image-toimage translation methods for data augmentation, our key insight is that while the translated images between domains may differ in styles, their predictions for the task should be consistent. We exploit this property and introduce a crossdomain consistency loss that enforces our adapted model to produce consistent predictions. Through extensive experimental results, we show that our method compares favorably against the state-of-the-art on a wide variety of unsupervised domain adaptation tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

Chen

Lin

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

312

199

View full text Add to dashboard Cite

show abstract

“…Since the source code for Super SloMo is not available, we implemented it ourselves. We replaced the flow computation network with PWC-Net [19] as it gives better performance, especially on videos with high resolution and large motion. For Gupta et al's approach, we use one of the scenes from their supplementary video for comparison.…”

Section: Resultsmentioning

confidence: 99%

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Paliwal

Kalantari

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Slow motion videos are becoming increasingly popular, but capturing high-resolution videos at extremely high frame rates requires professional high-speed cameras. To mitigate this problem, current techniques increase the frame rate of standard videos through frame interpolation by assuming linear motion between the existing frames. While this assumption holds true for simple cases with small motion, in challenging cases the motion is usually complex and this assumption is no longer valid. Therefore, they typically produce results with unnatural motion in these challenging cases. In this paper, we address this problem using two video streams as the input; an auxiliary video with high frame rate and low spatial resolution, providing temporal information, in addition to the standard main video with low frame rate and high spatial resolution. We propose a two-stage deep learning system consisting of alignment and appearance estimation that reconstructs high resolution slow motion video from the hybrid video input. For alignment, we propose to use a set of pre-trained and trainable convolutional neural networks (CNNs) to compute the flows between the missing frame and the two existing frames of the main video by utilizing the content of the auxiliary video frames. We then warp the existing frames using the flows to produce a set of aligned frames. For appearance estimation, we propose to combine the aligned and auxiliary frames using a context and occlusion aware CNN. We train our model on a set of synthetically generated hybrid videos and show high-quality results on a wide range of test scenes. We further demonstrate the practicality of our approach by showing the performance of our system on two real dual camera setups with small baseline.

show abstract

“…For the deblurring and optical flow modules, we take advantage of existing neural network architectures which have performed well in the past for the respective supervised learning tasks [9], [17], [21], [30], [32]. In particular, we adopt the single image deblurring network from Tao et al [32] and the dense optical flow estimation network PWC-Net from Sun et al [30]. We make the following modifications for the deblurring network for our particular problem: 1) We replace the deconvolution layer with bilinear upsampling followed by a 3x3 convolution to avoid upsampling artifacts.…”

Section: A Deblurring and Optical Flowmentioning

confidence: 99%

Self-Supervised Linear Motion Deblurring

Liu

Janai

Pollefeys

et al. 2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Motion blurry images challenge many computer vision algorithms, e.g., feature detection, motion estimation, or object recognition. Deep convolutional neural networks are stateof-the-art for image deblurring. However, obtaining training data with corresponding sharp and blurry image pairs can be difficult. In this paper, we present a differentiable reblur model for selfsupervised motion deblurring, which enables the network to learn from real-world blurry image sequences without relying on sharp images for supervision. Our key insight is that motion cues obtained from consecutive images yield sufficient information to inform the deblurring task. We therefore formulate deblurring as an inverse rendering problem, taking into account the physical image formation process: we first predict two deblurred images from which we estimate the corresponding optical flow. Using these predictions, we re-render the blurred images and minimize the difference with respect to the original blurry inputs. We use both synthetic and real dataset for experimental evaluations. Our experiments demonstrate that self-supervised single image deblurring is really feasible and leads to visually compelling results. Both the code and datasets are available at https://github.com/ethliup/SelfDeblur.

show abstract

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

Cited by 2,463 publications

References 57 publications

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Self-Supervised Linear Motion Deblurring

Contact Info

Product

Resources

About