Autoencoder-based background reconstruction and foreground segmentation with background noise estimation

Abstract

Even after decades of research, dynamic scene background reconstruction and foreground object segmentation are still considered as open problems due various challenges such as illumination changes, camera movements, or background noise caused by air turbulence or moving trees. We propose in this paper to model the background of a video sequence as a low dimensional manifold using an autoencoder and to compare the reconstructed background provided by this autoencoder with the original image to compute the foreground/background segmentation masks. The main novelty of the proposed model is that the autoencoder is also trained to predict the background noise, which allows to compute for each frame a pixel-dependent threshold to perform the background/foreground segmentation. Although the proposed model does not use any temporal or motion information, it exceeds the state of the art for unsupervised background subtraction on the CDnet 2014 and LASIESTA datasets, with a significant improvement on videos where the camera is moving.

Unsupervised Multi-object Segmentation Using Attention and Soft-argmax

“…It appears however that the interaction between these two models during training is a very challenging issue, because of the competition between them to reconstruct the image. We handle this problem by training the background model before the foreground model: We observe that the AE-NE model [40], which is dedicated to dynamic background reconstruction and segmentation, is trained without any foreground reconstruction module, and is able to perform an accurate background reconstruction and segmentation not only on videos, but also on frame sequences which are not organized as videos. We then use this model as a pre-trained separate module: This background model is first trained independently from the other parts of the model, and the weights of this background model are then frozen during the training of the foreground model which is described below.…”

“…3 on ObjectsRoom, 6 on ShapeStacks and 10 on CLEVRTEX). On CLEVR, which shows a fixed background, we reduce the number of background training iterations from 500 000 to 2500, as recommended in the AE-NE paper [40], and decrease the fixed background accuracy threshold τ since the background reconstruction is far more accurate when the background is fixed. We use isotropic scaling since all objects have similar widths and heights in these datasets.…”