SLAMP: Stochastic Latent Appearance and Motion Prediction

Akan, Adil Kaan; Erdem, Erkut; Erdem, Aykut; Güney, Fatma

doi:10.1109/iccv48922.2021.01446

Cited by 21 publications

(18 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent methods including SLAMP [1], Improved-VRNN [6], and our models clearly outperform SVG [9] and SRVP [11] in terms of all three metrics. Improved-VRNN [6] 1 achieves that by using six times more param- eters compared to our models (57M vs. 308M).…”

Section: Quantitative Resultsmentioning

confidence: 86%

“…We trained and evaluated stochastic video prediction methods on KITTI and Cityscapes by using the same number of conditioning frames including SVG [9], SRVP [11], SLAMP [1], and Improved-VRNN [6]. We optimized their architectures for a fair comparison (see Appendix).…”

Section: Quantitative Resultsmentioning

confidence: 99%

“…SLAMP [1] achieves the top-performing results on Cityscapes in terms of PSNR and SSIM, by explicitly modelling the motion history. The separation between the pixel and the motion space is achieved by learning a separate distribution for each with a slight increase in complexity compared to SVG.…”

Section: Quantitative Resultsmentioning

confidence: 99%

“…Decomposition: State-space model SRVP [11] learns a content variable from the first few frames which remains unchanged while predicting the future frames. As shown in SLAMP [1], the content variable in SRVP cannot handle changes in the background. In addition to pixel space, SLAMP separately models changes in motion and keeps track of a motion history.…”

Section: Related Workmentioning

confidence: 99%

“…Earlier methods encode the dynamics of a video in stochastic latent variables which are then decoded to predict future frames [9]. A recent method [1] proposes to incorporate the motion history by explicitly predicting motion between consecutive frames. In particular, they learn two separate distributions representing the changes in the pixel space and the motion space.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Stochastic Video Prediction with Structure and Motion

Akan¹,

Sadra²,

Erdem³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

While stochastic video prediction models enable future prediction under uncertainty, they mostly fail to model the complex dynamics of real-world scenes. For example, they cannot provide reliable predictions for scenes with a moving camera and independently moving foreground objects in driving scenarios. The existing methods fail to fully capture the dynamics of the structured world by only focusing on changes in pixels. In this paper, we assume that there is an underlying process creating observations in a video and propose to factorize it into static and dynamic components. We model the static part based on the scene structure and the ego-motion of the vehicle, and the dynamic part based on the remaining motion of the dynamic objects. By learning separate distributions of changes in foreground and background, we can decompose the scene into static and dynamic parts and separately model the change in each. Our experiments demonstrate that disentangling structure and motion helps stochastic video prediction, leading to better future predictions in complex driving scenarios on two real-world driving datasets, KITTI and Cityscapes.

show abstract

Section: Quantitative Resultsmentioning

confidence: 86%

Section: Quantitative Resultsmentioning

confidence: 99%

Section: Quantitative Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations