4D Visualization of Dynamic Events From Unconstrained Multi-View Videos

Bansal, Aayush; Vo, Minh; Sheikh, Yaser; Ramanan, Deva; Narasimhan, Srinivasa G.

doi:10.1109/cvpr42600.2020.00541

Cited by 63 publications

(43 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The recent neural rendering techniques bring huge potential for compelling photo-realistic free-viewpoint video generation via neural view blending [Meshry et al 2019a;Thies et al 2018] or neural scene modeling [Bemana et al 2020;Mildenhall et al 2020a;Wu et al 2020b]. Open4D [Bansal et al 2020] generates a free-viewpoint video enabling occlusion removal and time-freezing effects using around 15 mobile cameras, which is similar to our work. Such datadriven approaches get rid of the heavy reliance on reconstruction accuracy or the extremely dense capture setting.…”

Section: Introductionsupporting

confidence: 73%

“…However, these approaches above still cannot provide a wide-range free-viewing of a large dynamic scene, let alone editing various dynamic entities. To model dynamic scenes, the prior works [Bansal et al 2020;Carranza et al 2003;Lipski et al 2010;Zitnick et al 2004] require multi-view, time-synchronized videos as input for rendering various space-time visual effects. Zitnick et al [Zitnick et al 2004] use depth maps estimated from multi-view stereo to guide viewpoint interpolation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Generating free-viewpoint videos is critical for immersive VR/AR experience, but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper, we propose the first approach for editable free-viewpoint video generation for large-scale view-dependent dynamic scenes using only 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity, including the environment itself, is formulated into a spatiotemporal coherent neural layered radiance representation called ST-NeRF. Such a layered representation supports manipulations of the dynamic scene while still supporting a wide free viewing experience. In our ST-NeRF, we represent the dynamic entity/layer as a continuous function, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further

show abstract

Section: Introductionsupporting

confidence: 73%

Section: Related Workmentioning

confidence: 99%

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…A key advantage of our framework is that it provides 3D representations readily integrated modern neural rendering methods, which typically requires 3D and/or camera pose ground truth annotation at training time [14,48,4,72,63]. This usually implies a 2-stage process with running Structurefrom-Motion (SfM) as precursor to training.…”

Section: Introductionmentioning

confidence: 99%

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

Lai

Liu

Efros

et al. 2021

Preprint

View full text Add to dashboard Cite

Videostructure + motion … …Figure 1: Video Autoencoder: raw input video is automatically disentangled into 3D scene structure and camera trajectory. To reconstruct the original video, the camera transformation is applied to the 3D structure feature and then decoded back to pixels. Without any fine-tuning, the model generalizes to unseen videos and enables tasks such as novel view synthesis, pose estimation, and "video following".

show abstract

“…Consequently, these methods rely on high-quality 3D reconstruction or dense perpixel correspondence, which requires dense multi-views. Recent work [5,38] combines classical methods with datadriven approaches by learning to correct the warped views of classical methods. The sequential pipeline in these methods [5,38] do not allow end-to-end learning.…”

Section: Introductionmentioning

confidence: 99%

“…Recent work [5,38] combines classical methods with datadriven approaches by learning to correct the warped views of classical methods. The sequential pipeline in these methods [5,38] do not allow end-to-end learning.…”

Section: Introductionmentioning

confidence: 99%