IM-Net for High Resolution Video Frame Interpolation

Peleg, Tomer; Szekely, Pablo; Sabo, Doron; Sendik, Omry

doi:10.1109/cvpr.2019.00250

Cited by 90 publications

(69 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, deep neural networks have shown excellent performance in optical flow estimation [6], [7], [25]. Given two consecutive color images C t−1 and C t+1 , video interpolation [1], [8], [14] aims to generate the intermediate color image C t using a bidirectional optical flow:…”

Section: A Intermediate Depth Map Interpolationmentioning

confidence: 99%

See 1 more Smart Citation

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Liu

Liao

Lin

et al. 2020

Sensors

View full text Add to dashboard Cite

LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10Hz) and have been widely applied in the field of autonomous driving and UAV. However, the camera with a higher frequency (around 20Hz) has to be decreased so as to match with LiDAR in a multisensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of Li-DAR sensors. PLIN can generate temporally and spatially highquality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guided by consecutive sparse depth maps and motion relationship. We also propose a refined interpolation stage guided by the realistic scene. Using this coarse-to-fine cascade structure, our method can progressively perceive multi-modal information and generate accurate intermediate point clouds.To the best of our knowledge, this is the first deep framework for Pseudo-LiDAR point cloud interpolation, which shows appealing applications in navigation systems equipped with LiDAR and cameras. Experimental results demonstrate that PLIN achieves promising performance on the KITTI dataset, significantly outperforming the traditional interpolation method and the state-of-the-art video interpolation technique.

show abstract

Section: A Intermediate Depth Map Interpolationmentioning

confidence: 99%

“…This technique has attracted more attentions due to the increasing demand for high-quality slow-motion videos. For example, Peleg et al [14] formulated the interpolated motion estimation problem as classification rather than regression. This method achieves real-time temporal interpolation for high resolution videos.…”

Section: Introductionmentioning

confidence: 99%

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Liu

Liao

Lin

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…During the warping process, relative monocular depth is estimated so that closer objects contribute more during flow projection. Moreover, Peleg et al [37] focuses on real-time processing and a large receptive field. They extract a low-resolution feature from multi-scale architecture to acquire vertical and horizontal motion vector field.…”

Section: Frame Interpolationmentioning

confidence: 99%

“…It is used for training and testing TOF [52], DAIN [2]. IM-Net [37] uses super-resolved Vimeo sequences to evaluate the interpolation performance for large motion. SlowFlow [18] are high quality dataset for optical flow.…”

Section: Frame Interpolation Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

Nah

Kim

Han

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Videos contain various types and strengths of motions that may look unnaturally discontinuous in time when the recorded frame rate is low. This paper reviews the first AIM challenge on video temporal super-resolution (frame interpolation) with a focus on the proposed solutions and results. From low-frame-rate (15 fps) video sequences, the challenge participants are asked to submit higher-framerate (60 fps) video sequences by estimating temporally intermediate frames. We employ the REDS VTSR dataset derived from diverse videos captured in a hand-held camera for training and evaluation purposes. The competition had 62 registered participants, and a total of 8 teams competed in the final testing phase. The challenge winning methods achieve the state-of-the-art in video temporal superresolution. Resolution Members: Seungjun Nah 1

show abstract

Native and Non‐Native English Speaking Teachers in China: Perceptions and PracticesZhengHuang. Shanghai, China: Shanghai Jiao Tong University Press and Singapore: Springer, 2018. Pp. xv + 192.

2018

TESOL Quarterly

View full text Add to dashboard Cite

In this paper, we consider the problem of disease diagnosis. Unlike the conventional learning paradigm that treats labels independently, we propose a knowledge-enhanced framework, that enables training visual representation with the guidance of medical domain knowledge. In particular, we make the following contributions: First, to explicitly incorporate experts' knowledge, we propose to learn a neural representation of medical knowledge graph via contrastive learning, implicitly establishing relations between different medical concepts. Second, while training the visual encoder, we keep the parameters of the knowledge encoder frozen and propose to learn a set of prompt vectors for efficient adaptation. Third, we adopt a Transformer-based disease-query module for cross-model fusion, which naturally enables explainable diagnosis results via cross attention. To validate the effectiveness of our proposed framework, we conduct thorough experiments on three x-ray imaging datasets across different anatomy structures, showing our model can exploit the implicit relations between diseases/findings, thus is beneficial to the commonly encountered problem in the medical domain, namely, long-tailed and zero-shot recognition, which conventional methods either struggle or completely fail to realize.

show abstract

IM-Net for High Resolution Video Frame Interpolation

Cited by 90 publications

References 19 publications

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

Native and Non‐Native English Speaking Teachers in China: Perceptions and PracticesZhengHuang. Shanghai, China: Shanghai Jiao Tong University Press and Singapore: Springer, 2018. Pp. xv + 192.

Contact Info

Product

Resources

About