Video Enhancement with Task-Oriented Flow

Xue, Tao; Chen, Baian; Wu, Jiajun; Wei, Donglai; Freeman, William T.

doi:10.1007/s11263-018-01144-2

Cited by 1,252 publications

(1,189 citation statements)

References 51 publications

Supporting

Mentioning

1,188

Contrasting

Unclassified

Order By: Relevance

“…We use the Vimeo90K dataset [40] to train our model. The Vimeo90K dataset has 64,612 septuplets for training, where each septuplet contains 7 consecutive video frames at a size of 256 × 448 pixels.…”

Section: Training Datasetmentioning

confidence: 99%

“…We first compare our method with the state-of-the-art SISR method EDSR [28], task-oriented video super-resolution method ToFlow-SR [40], conventional RefSR patchmatch (PM) [6], and the state-of-the-art learning-based RefSR CrossNet [43]. To be fair, we have retrained CrossNet with our dataset following the training strategy suggested in [43].…”

Section: Performance Comparison Using Simulation Datamentioning

confidence: 99%

“…Frame interpolation: Our AWnet can also be used to interpolate missing intermediate frames (usually at high spatial resolution) with the help from another LSR-HFR input. Such frame interpolation is also supported by optical flow based methods, such as ToFlow-Intp in [40]. We use the third and the fifth frames from the testing septuplets to interpolate missing fourth frame.…”

Section: Super-resolutionmentioning

confidence: 99%

See 2 more Smart Citations

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Asif

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects (such as noise regularization, camera parallax, exposure time, multiscale synthesis, etc) of our system to fully understand its capability for potential applications. respectively. His current research focuses on the next-generation video coding, energy-efficient communication, gigapixel streaming and deep learning. He is a co-recipient of 2018 ACM SIGCOMM Student Research Competition Finalist, 2018 PCM

show abstract

Section: Training Datasetmentioning

confidence: 99%

Section: Performance Comparison Using Simulation Datamentioning

confidence: 99%

Section: Super-resolutionmentioning

confidence: 99%

See 1 more Smart Citation

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Asif

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Recently, several works were proposed to exploit the spatiotemporal information from neighboring frames. Xue et al [42] designed a neural network with a motion estimation component and a video processing component, and utilized a joint training strategy to handle various low-level vision tasks . Lu et al [26] further incorporated quantized prediction residual in compressed code streams as strong prior knowledge, and proposed a deep Kalman filter network (DKFN) to utilize the spatiotemporal information from the preceding frames of the target frame.…”

Section: Video Compression Artifact Reductionmentioning

confidence: 99%

“…In addition to compression artifact removal, spatiotemporal correlation mining is also a hot topic in other video quality enhancement tasks, such as video super resolution (VSR). [4,18,19,23,32,37,42] estimated optical flow and warped several frames to capture the hidden spa- tiotemporal dependency for VSR. Although these methods work well, they rely heavily on the accuracy of motion estimation.…”

Section: Video Compression Artifact Reductionmentioning

confidence: 99%

Non-Local ConvLSTM for Video Compression Artifact Reduction

Gao²,

Tian

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Video compression artifact reduction aims to recover high-quality videos from low-quality compressed videos. Most existing approaches use a single neighboring frame or a pair of neighboring frames (preceding and/or following the target frame) for this task. Furthermore, as frames of high quality overall may contain low-quality patches, and high-quality patches may exist in frames of low quality overall, current methods focusing on nearby peak-quality frames (PQFs) may miss high-quality details in low-quality frames. To remedy these shortcomings, in this paper we propose a novel end-to-end deep neural network called non-local ConvLSTM (NL-ConvLSTM in short) that exploits multiple consecutive frames. An approximate non-local strategy is introduced in NL-ConvLSTM to capture global motion patterns and trace the spatiotemporal dependency in a video sequence. This approximate strategy makes the non-local module work in a fast and low space-cost way. Our method uses the preceding and following frames of the target frame to generate a residual, from which a higher quality frame is reconstructed. Experiments on two datasets show that NL-ConvLSTM outperforms the existing methods.

show abstract

Achromatic and Resolution Enhancement Light Field Deep Neural Network for ZnO Nematic Liquid Crystal Microlens Array

Li,

Chen

et al. 2023

Advanced Photonics Research

View full text Add to dashboard Cite

Nematic liquid‐crystal microlens arrays (LC‐MLAs) often exhibit chromatic aberration and low resolution, severely compromising their optical imaging quality. This study proposes an achromatic and resolution enhancement light field (ARELF) deep neural network (DNN) to address these issues. The training set is constructed by incorporating LC‐MLA characteristics’ degradation, retrofitting the vimeo90k dataset. The network's hidden layer is trained to learn about chromatic aberration and low resolution of LC‐MLA while extracting imaging features and fusing the information of complementary features of a light field under varying voltages. The loss function includes both chromatic aberration and overall resolution. The light field images of ZnO LC‐MLA under seven consecutive voltages are used as input to test the proposed DNN model. After experimental verification, the proposed model effectively eliminates chromatic aberration while enhancing the spatial and temporal resolution of LC‐MLA. This novel network can be utilized to optimize the design process of LC‐MLA and significantly improve its imaging performance.

show abstract

Video Enhancement with Task-Oriented Flow

Abstract: t=1 t=2 t=3 (I-b) EpicFlow (I-d) Task-oriented Flow (I-c) Interp. by EpicFlow (I-e) Interp. by Task-oriented Flow

Cited by 1,252 publications

References 51 publications

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Non-Local ConvLSTM for Video Compression Artifact Reduction

Achromatic and Resolution Enhancement Light Field Deep Neural Network for ZnO Nematic Liquid Crystal Microlens Array

Contact Info

Product

Resources

About