Progressive Temporal Feature Alignment Network for Video Inpainting

Zou, Xueyan; Yang, Linjie; Liu, Ding; Lee, Yong Jae

doi:10.1109/cvpr46437.2021.01618

Cited by 48 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are a few recent methods than build on STTN to generate higher resolution inpainted textures. These include introducing an Aggregated Contextual-Transformation GAN (AOT-GAN) Zeng et al (2021), combining 3D CNNs with a temporal shift and align module Zou et al (2021), and introducing a Deformable Alignment and Pyramid Context Completion Network with temporal attention Wu et al (2021). Addition-ally, more complex occlusions can be handled with a Decoupled Spatial-Temporal Transformer with a hierarchical encoder Liu et al (2021).…”

Section: Temporal Video Inpaintingmentioning

confidence: 99%

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

View full text Add to dashboard Cite

Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames where they are not present in the same location. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner that relies on automatic detection of specular highlights. System evaluations show significant improvements to traditional methods through direct comparison as well as other machine learning techniques through an ablation study that depicts the importance of the network's temporal and transfer learning components. The generalizability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of specular highlight inpainting on these tasks in a novel comprehensive analysis.

show abstract

Section: Temporal Video Inpaintingmentioning

confidence: 99%

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some methods [7,23,28,50] employing 3D convolution and attention usually yield temporally inconsistent results due to the limited temporal receptive fields. To generate more temporal coherence results, many works [23,67] regard optical flows as strong priors for video inpainting and incorporate them into the network. However, directly computing optical flows between images within invalid regions is extremely difficult as these regions themselves become occlusion factors, restricting the performance.…”

Section: Related Workmentioning

confidence: 99%

“…To address these flaws, in this paper, we carefully design three trainable modules, including (1) flow completion, (2) feature propagation, and (3) content hallucination modules which simulate corresponding stages in flow-based methods and further constitute an End-to-End framework for Flow-Guided Video Inpainting (E 2 FGVI). Such close collaboration between the three modules alleviates the excessive dependence of intermediate results in the previously independently developed system [17,23,26,57,67] and works in a more efficient manner.…”

Section: Introductionmentioning

confidence: 99%

Towards An End-to-End Framework for Flow-Guided Video Inpainting

Li¹,

Lu²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Optical flow, which captures motion information across frames, is exploited in recent video inpainting methods through propagating pixels along its trajectories. However, the hand-crafted flow-based processes in these methods are applied separately to form the whole inpainting pipeline. Thus, these methods are less efficient and rely heavily on the intermediate results from earlier stages. In this paper, we propose an End-to-End framework for Flow-Guided Video Inpainting (E 2 FGVI) through elaborately designed three trainable modules, namely, flow completion, feature propagation, and content hallucination modules. The three modules correspond with the three stages of previous flowbased methods but can be jointly optimized, leading to a more efficient and effective inpainting process. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods both qualitatively and quantitatively and shows promising efficiency. The code is available at https://github.com/MCG-NKU/E2FGVI.

show abstract

“…In general, direct synthesis methods adopt convolution-based [3,4,7,23] and attention-based [13,22] networks. They usually take corrupted frames and corresponding pixel-wise masks as input and directly output completed frames.…”

Section: Introductionmentioning

confidence: 99%

“…However, due to the computational complexity, these methods have relatively small temporal windows resulting in only a few reference frames to be used (up to 10 frames). Serious temporal inconsistencies can occur if some key frames The rear camel is originally shown in all frames, but it disappears in the result of [22,23] due to the small temporal window. (b) Wrong pixels are propagated, and the brightness inconsistency issue occurred in [6,20].…”

Section: Introductionmentioning

confidence: 99%

Error Compensation Framework for Flow-Guided Video Inpainting

Kang¹,

Oh²,

Kim³

2022

Preprint

View full text Add to dashboard Cite

The key to video inpainting is to use correlation information from as many reference frames as possible. Existing flow-based propagation methods split the video synthesis process into multiple steps: flow completion → pixel propagation → synthesis. However, there is a significant drawback that the errors in each step continue to accumulate and amplify in the next step. To this end, we propose an Error Compensation Framework for Flow-guided Video Inpainting (ECFVI), which takes advantage of the flow-based method and offsets its weaknesses. We address the weakness with the newly designed flow completion module and the error compensation network that exploits the error guidance map. Our approach greatly improves the temporal consistency and the visual quality of the completed videos. Experimental results show the superior performance of our proposed method with the speed up of ×6, compared to the state-of-the-art methods. In addition, we present a new benchmark dataset for evaluation by supplementing the weaknesses of existing test datasets.

show abstract

Progressive Temporal Feature Alignment Network for Video Inpainting

Cited by 48 publications

References 27 publications

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Towards An End-to-End Framework for Flow-Guided Video Inpainting

Error Compensation Framework for Flow-Guided Video Inpainting

Contact Info

Product

Resources

About