Interactive Scene Flow Editing for Improved Image-based Rendering and Virtual Spacetime Navigation

Abstract

High-quality stereo and optical flow maps are essential for a multitude of tasks in visual media production, e.g. virtual camera navigation, disparity adaptation or scene editing. Rather than estimating stereo and optical flow separately, scene flow is a valid alternative since it combines both spatial and temporal information and recently surpassed the former two in terms of accuracy. However, since automated scene flow estimation is non-accurate in a number of situations, resulting rendering artifacts have to be corrected manually in each output frame, an elaborate and time-consuming task. We propose a novel workflow to edit the scene flow itself, catching the problem at its source and yielding a more flexible instrument for further processing. By integrating user edits in early stages of the optimization, we allow the use of approximate scribbles instead of accurate editing, thereby reducing interaction times. Our results show that editing the scene flow improves the quality of visual results considerably while requiring vastly less editing effort.

Surface tracking assessment and interaction in texture space

“…While Zhang et al [6] directly work on the maps by letting the user correct existing disparity maps on key frames, other approaches work in the image domain and use sparse scribbles on the 2D images to define depth layers, using them as soft constraints in a global optimization framework which propagates them into per-pixel depth maps through the whole image or video sequence [11,12]. Similarly, other approaches use simple paint strokes to let the user set smoothness, discontinuity, and depth ordering constraints in a variational optimization framework [4,5,13].…”

“…However, while increasingly being discussed in the research community [4][5][6], visual effects artists have not yet adopted user interaction with dense optical flowbased estimation methods.…”