Weakly Supervised Object Localization with Large Fisher Vectors

Abstract

We propose a novel method for learning object localization models in a weakly supervised manner, by employing images annotated with object class labels but not with object locations. Given an image, the learned model predicts both the presence of the object class in the image and the bounding box that determines the object location. The main ingredients of our method are a large Fisher vector representation and a sparse classification model enabling efficient evaluation of patch scores. The method is able to reliably detect very small objects with some intra-class variation in reasonable time. Experimental validation has been performed on a public dataset and we report localization performance comparable to strongly supervised approaches.

Patch-Level Spatial Layout for Classification and Weakly Supervised Localization

“…Here we require the classification score gradient in order to be able to identify the positive patches. We observe that the combination of the group-sparsity and spatial layout model achieves the best classification AP (around 81%), which is 4 pp better than the appearance-based counterpart (row 7) and more than 9 pp better than [21] (row 2). The group-sparse model identifies only 7 visual words, so there are only 49 possible pairs to consider in the spatial model.…”

“…The results show that the group-sparse model outperforms the 2 regularized model for 7 percentage points (pp). In comparison with the 1 regularized model [21], the group-sparse model is 17 times more sparse and achieves comparable AP. This implies substantial performance advantage in terms of execution time.…”

“…The results stress out the importance of non-linear normalizations (rows 4 and 5) as they increase the AP by 5 pp w.r.t. the weakly supervised baseline [21] (row 2). Further, by using the gradient approximation (row 5) instead of the direct patch contribution f (X) − f (X − x) (row 4), we obtain a comparable AP but pay the price by increasing the p miss for 5 pp points.…”

“…Due to generative front-end, we have a better sharing potential than pure discriminative approaches used in [27]. In contrast with [21], we use block-sparsity [20], the normalized score gradient and the spatial model of the pairwise layout.…”

“…We have previously shown [21] that traffic signs can be localized by a sparse linear model trained on non-normalized Fisher vectors (FV) of entire images. In this paper we present two contributions which further improve these results.…”