Interactive Videos: Plausible Video Editing using Sparse Structure Points

Abstract

a) Input video (b) Sparse structure points (c) Input video (d) Sparse structure points Figure 1: Starting from raw video sequences (a,c), we extract corresponding sets of sparse structure points (b,d), which are then used to enable direct object level video edits (bottom row). (Please refer to the supplementary for the video sequences.) AbstractVideo remains the method of choice for capturing temporal events. However, without access to the underlying 3D scene models, it remains difficult to make object level edits in a single video or across multiple videos. While it may be possible to explicitly reconstruct the 3D geometries to facilitate these edits, such a workflow is cumbersome, expensive, and tedious. In this work, we present a much simpler workflow to create plausible editing and mixing of raw video footage using only sparse structure points (SSP) directly recovered from the raw sequences. First, we utilize user-scribbles to structure the point representations obtained using structure-from-motion on the input videos. The resultant structure points, even when noisy and sparse, are then used to enable various video edits in 3D, including view perturbation, keyframe animation, object duplication and transfer across videos, etc. Specifically, we describe how to synthesize object images from new views adopting a novel image-based rendering technique using the SSPs as proxy for the missing 3D scene information. We propose a structure-preserving image warping on multiple input frames adaptively selected from object video, followed by a spatio-temporally coherent image stitching to compose the final object image. Simple planar shadows and depth maps are synthesized for objects to generate plausible video sequence mimicking real-world interactions. We demonstrate our system on a variety of input videos to produce complex edits, which are otherwise difficult to achieve.

AI Video Editing: a Survey

“…It could generates interactively controllable video sprites that behave and appear like professional players. Chang et al [194] present a system that support object-level video editing. It segement objects and generates alpha mattes, and estimates 3D scene information based on structure from-motion (SfM) algorithm [195].…”

AI Video Editing: a Survey

“…It could generate interactively controllable video sprites that behave and appear like professional players. Chang et al [194] present a system that supports object-level video editing. It segments objects and generates alpha mattes, and estimates 3D scene information based on structure-from-motion (SfM) algorithm [195].…”

AniCode: authoring coded artifacts for network-free personalized animations

“…Similar work includes manipulating existing objects interactively with partial scene reconstructions based on cuboid proxies [Zheng et al 2012]. Authoring time-based media for AR has also become possible by mixing raw video footage using sparse structure points [Chang et al 2016]. Recent AR devices such as Microsoft HoloLens provide the possibility of interactive scene editing in the real world based on data from the sensor [Yue et al 2017].…”