FILM: Frame Interpolation for Large Motion

Reda, Fitsum A.; Kontkanen, Janne; Tabellion, Eric; Sun, Deqing; Pantofaru, Caroline; Curless, Brian

doi:10.1007/978-3-031-20071-7_15

Cited by 75 publications

(41 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extending to Frame Interpolation. Our method can be potentially modified and improved for frame interpolation using only two input images [Reda et al 2022]. As a proof of concept, we take two human portrait images of the same person and deploy our method on those images.…”

Section: 24mentioning

confidence: 99%

VIINTER: View Interpolation with Implicit Neural Representations of Images

Feng

Jabbireddy

Varshney

2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

Figure 1: We propose a new method for view interpolation through implicit neural representations (INR) of images. After each image is randomly assigned a code vector 𝑧, the codes are then jointly trained with the neural network to produce the RGB color given coordinate (𝑥, 𝑦). With standard training, the INR fails to decode coherent images from new codes interpolated by two trained codes, but our method enables smooth transition between two known viewpoints. Contrary to common methods for view interpolation, our method does not use 3D structure, camera poses, or pixel correspondence during training.

show abstract

Section: 24mentioning

confidence: 99%

VIINTER: View Interpolation with Implicit Neural Representations of Images

Feng

Jabbireddy

Varshney

2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

show abstract

“…Moreover, instead of predicting target texture directly from deep feature like most other VFI algorithms [5], [8], [14], [11], [12], [20], we propose a new context synthesis module (CSM) to simplify the frame synthesis in each level by borrowing existing texture from adjacent input image pyramids, which means that decoder D l of PMCRNet only needs to predict easier intermediate optical flow F l−1 t→0 , F l−1 t→1 , one-channel occlusion merge mask M l−1 and three-channel image residual R l−1 as shown in Fig. 1.…”

Section: B Joint Refinement Decodermentioning

confidence: 99%

“…Based on above analysis, a motion context joint refinement based PMCRNet has been built for VFI. To optimize this network, we employ the Charbonnier loss [29] ρ to replace the L 1 loss like many existing methods [5], [6], [11], [12], [20]. Besides, inspired by the robustness of the census loss L cen in unsupervised optical flow estimation [30], [31], we add it as a complementary loss term, which calculates the soft Hamming distance between census-transformed image patches of size 7×7.…”

Section: Loss Functionmentioning

confidence: 99%

“…Whereas, this framework can not deal with large motion cases well and the interpolated frames tend to have sudden change at patch edges. Recently, FILM [20] builds a single unified network distinguished by a shared multi-scale feature extractor for both motion estimation and frame synthesis. However, they directly generate target frame from deep feature, which suffers from large computation cost and blur artifacts.…”

Section: Introductionmentioning

confidence: 99%

“…It first extracts pyramid features from two input frames, and then estimates intermediate optical flow and image context in a coarse-to-fine manner, which can deal with the large displacement challenge. Different from others that directly generate target frame from deep feature [8], [14], [11], [12], [20], we simplify the frame synthesis in each scale by borrowing existing texture from adjacent input image pyramids, which means that each decoder only needs to refine easier intermediate optical flow, occlusion merge mask and image residual. Corresponding to this progressive refinement architecture, we further introduce a new annealed multi-scale reconstruction loss to provide better learning guidance, where it does not need additional pre-trained optical flow network,…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

Kong

Liu

Yang

2022

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Recently, flow-based frame interpolation methods have achieved great success by first modeling optical flow between target and input frames, and then building synthesis network for target frame generation. However, above cascaded architecture can lead to large model size and inference delay, hindering them from mobile and real-time applications. To solve this problem, we propose a novel Progressive Motion Context Refine Network (PMCRNet) to predict motion fields and image context jointly for higher efficiency. Different from others that directly synthesize target frame from deep feature, we explore to simplify frame interpolation task by borrowing existing texture from adjacent input frames, which means that decoder in each pyramid level of our PMCRNet only needs to update easier intermediate optical flow, occlusion merge mask and image residual. Moreover, we introduce a new annealed multi-scale reconstruction loss to better guide the learning process of this efficient PMCRNet. Experiments on multiple benchmarks show that proposed approaches not only achieve favorable quantitative and qualitative results but also reduces current model size and running time significantly.

show abstract

A novel, end‐to‐end framework for avoiding collisions between the patient's body and gantry in proton therapy

Yamazaki,

Terunuma,

Kato

et al. 2023

Medical Physics

View full text Add to dashboard Cite

BackgroundAdministration of external radiation therapy via proton therapy systems carries a risk of occasional collisions between the patient's body and gantry, which is increased by the snout placed near the patient for better dose distribution. Although treatment planning software (TPS) can simulate controlled collisions, the computed tomography (CT) data used for treatment planning are insufficient given that collisions can occur outside the CT imaging region. Thus, imaging the three‐dimensional (3D) surface outside the CT range and combining the data with those obtained by CT are essential for avoiding collisions.PurposeTo construct a prototype for 3D surface imaging and an end‐to‐end framework for preventing collisions between the patient's body and the gantry.MethodsWe obtained 3D surface data using a light sectioning method (LSM). By installing only cameras in front of the CT, we achieved LSM using the CT couch motion and preinstalled patient‐positioning lasers. The camera image contained both sagittal and coronal lines, which are unnecessary for LSM and were removed by deep learning. We combined LSM 3D surface data and original CT data to create synthetic Digital Imaging and Communications in Medicine (DICOM) data. Subsequently, we compared the TPS snout auto‐optimization using the original CT data with the synthetic DICOM data.ResultsThe mean positional error for LSM of the arms and head was 0.7 ± 0.8 and 0.8 ± 0.8 mm for axial and sagittal imaging, respectively. The TPS snout auto‐optimization indicated that the original CT data would cause collisions; however, the synthetic DICOM data prevented these collisions.ConclusionsThe prototype system's acquisition accuracy for 3D surface data was approximately 1 mm, which was sufficient for the collision simulation. The use of a TPS with collision avoidance can help optimize the snout position using synthetic DICOM data. Our proposed method requires no external software for collision simulation and can be integrated into the clinical workflow to improve treatment planning efficiency.

show abstract

FILM: Frame Interpolation for Large Motion

Cited by 75 publications

References 30 publications

VIINTER: View Interpolation with Implicit Neural Representations of Images

VIINTER: View Interpolation with Implicit Neural Representations of Images

Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

A novel, end‐to‐end framework for avoiding collisions between the patient's body and gantry in proton therapy

Contact Info

Product

Resources

About