Backpropagation Training for Fisher Vectors within Neural Networks

Abstract

Fisher-Vectors (FV) encode higher-order statistics of a set of multiple local descriptors like SIFT features. They already show good performance in combination with shallow learning architectures on visual recognitions tasks. Current methods using FV as a feature descriptor in deep architectures assume that all original input features are static. We propose a framework to jointly learn the representation of original features, FV parameters and parameters of the classifier in the style of traditional neural networks. Our proof of concept implementation improves the performance of FV on the Pascal Voc 2007 challenge in a multi-GPU setting in comparison to a default SVM setting. We demonstrate that FV can be embedded into neural networks at arbitrary positions, allowing end-to-end training with backpropagation.

End-to-end Learning of a Fisher Vector Encoding for Part Features in Fine-grained Recognition

“…Further details for the derivation of these gradients can be found in the supplementary material. Note that similar formulas are also provided by [36].…”

“…In contrast to the aforementioned methods that compute an FVE of local features extracted separately from the image, Wieshollek et al [36] and Tang et al [31] deploy the FVE directly in a neural network. As a result, the features are learned jointly with the parameters for both the classification and the mixture model.…”

“…In contrast to previous approaches that apply some kind of FVE together with CNNs [5,28,31,36], our method determines the underlying Gaussian mixture model (GMM) with an iterative EM algorithm using mini-batch updates of parameters. With our approach, the parameters of the GMM are estimated jointly with the parameters of the CNN in an end-to-end manner.…”