Second-Order Democratic Aggregation

Abstract

Aggregated second-order features extracted from deep convolutional networks have been shown to be effective for texture generation, fine-grained recognition, material classification, and scene understanding. In this paper, we study a class of orderless aggregation functions designed to minimize interference or equalize contributions in the context of second-order features and we show that they can be computed just as efficiently as their first-order counterparts and they have favorable properties over aggregation by summation. Another line of work has shown that matrix power normalization after aggregation can significantly improve the generalization of second-order representations. We show that matrix power normalization implicitly equalizes contributions during aggregation thus establishing a connection between matrix normalization techniques and prior work on minimizing interference. Based on the analysis we present γ-democratic aggregators that interpolate between sum (γ=1) and democratic pooling (γ=0) outperforming both on several classification tasks. Moreover, unlike power normalization, the γdemocratic aggregations can be computed in a low dimensional space by sketching that allows the use of very high-dimensional second-order features. This results in a state-of-the-art performance on several datasets.

Set-to-Sequence Methods in Machine Learning: A Review

“…AttSets improves the performance of previous, simpler pooling functions used for 3D object recognition. These include both first-order operators such as max(), average() and sum(), which do not have any trainable parameters, as well as higher-order statistical functions such as bilinear pooling (Lin et al, 2018), log-covariance pooling (Ionescu et al, 2015) and harmonized bilinear pooling (Yu et al, 2018), which have only few.…”

Two-Level Progressive Attention Convolutional Network for Fine-Grained Image Recognition

“…However, due to a very high dimension, the bilinear feature is difficult to be applied in practice. In order to improve the application ability of the B-CNNs, [17], [30], [31] reduced the parameter dimension of B-CNNs, and [18], [32], [33] made some variants based on B-CNNs to improve the accuracy of finegrained image recognition. These works have brought a lot of inspiration to our work.…”

ProLFA: Representative prototype selection for local feature aggregation

“…Although this kind of approach, such as FV [4], SV [8], VLAD [10], and VLAT [11], takes more advantage, they still suffer from the influence of pipeline mode, where the codebook learning is independent from feature aggregation. To address this issue, γ-democratic [29] exploited the relationship between democratic pooling and spectral normalization in the context of second-order features, and then proposed an aggregation approach in an end-to-end manner. In addition, based on shallow aggregation approaches, several neural network based aggregation methods have also been proposed.…”

“…We first test the performance of our ProLFA as in Section 6.4, where the prototype size is 4096. Then, we compare our method with a recent approach named γ-democratic [29] and a neural network-based approach named FV+NN [30]. For γ-democratic [29], we directly adopt the aggregated features from their source code 6 with γ = 0.5, while FV+NN [30] is reproduced by following FV encoding layer with a Multi-Layer Perceptron without data augmentation but with bagging.…”