Information Entropy Based Feature Pooling for Convolutional Neural Networks

Wan, Wei; Chen, Jiansheng; Li, Tianpeng; Huang, Yiqing; Tian, Jingqi; Cheng, Yu; Xue, Youze

doi:10.1109/iccv.2019.00350

Cited by 29 publications

(15 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Moldovan et al (2020 ), the authors use the so-called transfer entropy between network nodes to guide backpropagation. Other works use statistics for feature extraction ( Finnegan and Song, 2017 ), feature pooling ( Wan et al, 2019 ), or network compression ( Wiedemann et al, 2019 ). Related to that is the research on the distribution of activations, which often treats all neurons as independent stochastic variables and has proven helpful for derivations of initialization schemes and methods to help with training ( Glorot and Bengio, 2010 ; He et al, 2015 ; Ioffe and Szegedy, 2015 ; Salimans and Kingma, 2016 ).…”

Section: Discussionmentioning

confidence: 99%

Studying the Evolution of Neural Activation Patterns During Training of Feed-Forward ReLU Networks

Hartmann

Franzen

Brodehl

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

The ability of deep neural networks to form powerful emergent representations of complex statistical patterns in data is as remarkable as imperfectly understood. For deep ReLU networks, these are encoded in the mixed discrete–continuous structure of linear weight matrices and non-linear binary activations. Our article develops a new technique for instrumenting such networks to efficiently record activation statistics, such as information content (entropy) and similarity of patterns, in real-world training runs. We then study the evolution of activation patterns during training for networks of different architecture using different training and initialization strategies. As a result, we see characteristic- and general-related as well as architecture-related behavioral patterns: in particular, most architectures form bottom-up structure, with the exception of highly tuned state-of-the-art architectures and methods (PyramidNet and FixUp), where layers appear to converge more simultaneously. We also observe intermediate dips in entropy in conventional CNNs that are not visible in residual networks. A reference implementation is provided under a free license1.

show abstract

Section: Discussionmentioning

confidence: 99%

Studying the Evolution of Neural Activation Patterns During Training of Feed-Forward ReLU Networks

Hartmann

Franzen

Brodehl

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…The idea of entropy has been introduced to convolutional neural networks for different purposes. For instance, in [19] the authors use information entropy for semantic-aware feature pooling. In [20], an entropy measure is employed for the quantization of different deep learning models, including CNNs.…”

Section: Related Workmentioning

confidence: 99%

Using Non-Additive Entropy to Enhance Convolutional Neural Features for Texture Recognition

Florindo

Metze

2021

Entropy

View full text Add to dashboard Cite

Here we present a study on the use of non-additive entropy to improve the performance of convolutional neural networks for texture description. More precisely, we introduce the use of a local transform that associates each pixel with a measure of local entropy and use such alternative representation as the input to a pretrained convolutional network that performs feature extraction. We compare the performance of our approach in texture recognition over well-established benchmark databases and on a practical task of identifying Brazilian plant species based on the scanned image of the leaf surface. In both cases, our method achieved interesting performance, outperforming several methods from the state-of-the-art in texture analysis. Among the interesting results we have an accuracy of 84.4% in the classification of KTH-TIPS-2b database and 77.7% in FMD. In the identification of plant species we also achieve a promising accuracy of 88.5%. Considering the challenges posed by these tasks and results of other approaches in the literature, our method managed to demonstrate the potential of computing deep learning features over an entropy representation.

show abstract

“…where F m i and F d i stand for the multi-scale mean feature and std feature of frame i. However, it may not be feasible to concatenate the two pooled features straightforwardly for quality regression, due to the high relevance of F m i with the semantic information [46]. As a result, the learned model tends to overfit to the specific scenes in the training set.…”

Section: A Attention Based Multi-scale Feature Extractionmentioning

confidence: 99%

Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

Chen¹,

Zhu²,

Guo³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this work, we propose a no-reference video quality assessment method, aiming to achieve high-generalization capability in cross-content, -resolution and -frame rate quality prediction. In particular, we evaluate the quality of a video by learning effective feature representations in spatial-temporal domain. In the spatial domain, to tackle the resolution and content variations, we impose the Gaussian distribution constraints on the quality features. The unified distribution can significantly reduce the domain gap between different video samples, resulting in more generalized quality feature representation. Along the temporal dimension, inspired by the mechanism of visual perception, we propose a pyramid temporal aggregation module by involving the short-term and long-term memory to aggregate the frame-level quality. Experiments show that our method outperforms the state-of-the-art methods on cross-dataset settings, and achieves comparable performance on intra-dataset configurations, demonstrating the high-generalization capability of the proposed method.

show abstract

Information Entropy Based Feature Pooling for Convolutional Neural Networks

Cited by 29 publications

References 24 publications

Studying the Evolution of Neural Activation Patterns During Training of Feed-Forward ReLU Networks

Studying the Evolution of Neural Activation Patterns During Training of Feed-Forward ReLU Networks

Using Non-Additive Entropy to Enhance Convolutional Neural Features for Texture Recognition

Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

Contact Info

Product

Resources

About