A Spatiotemporal Oriented Energy Network for Dynamic Texture Recognition

Abstract

This paper presents a novel hierarchical spatiotemporal orientation representation for spacetime image analysis. It is designed to combine the benefits of the multilayer architecture of ConvNets and a more controlled approach to spacetime analysis. A distinguishing aspect of the approach is that unlike most contemporary convolutional networks no learning is involved; rather, all design decisions are specified analytically with theoretical motivations. This approach makes it possible to understand what information is being extracted at each stage and layer of processing as well as to minimize heuristic choices in design. Another key aspect of the network is its recurrent nature, whereby the output of each layer of processing feeds back to the input. To keep the network size manageable across layers, a novel cross-channel feature pooling is proposed. The multilayer architecture that results systematically reveals hierarchical image structure in terms of multiscale, multiorientation properties of visual spacetime. To illustrate its utility, the network has been applied to the task of dynamic texture recognition. Empirical evaluation on multiple standard datasets shows that it sets a new state-of-the-art.

Beyond the feedforward sweep: feedback computations in the visual cortex

“…Interestingly, several successful approaches to vision involve such feedforward architectures, where the same weights are reused recursively several times to increase the depth of visual processing. Indeed, the first texture discrimination algorithms were recursive, 36 and related ideas have also been applied to the recognition of dynamic texture 37 . Similarly, a hierarchical extension of the classic wavelet transform where the transform is applied recursively (also known as the scattering transform) has been shown to yield significant improvements in texture categorization 38 .…”

“…Indeed, the first texture discrimination algorithms were recursive, 36 and related ideas have also been applied to the recognition of dynamic texture. 37 Similarly, a hierarchical extension of the classic wavelet transform where the transform is applied recursively (also known as the scattering transform) has been shown to yield significant improvements in texture categorization. 38 Such recursive architectures can be implemented by RNNs within a single fully recurrent layer of processing.…”

Provably Scale-Covariant Continuous Hierarchical Networks Based on Scale-Normalized Differential Expressions Coupled in Cascade

“…For example, if we in a discrete implementation discretize the angles ϕ ∈ [0, π[ into M discrete spatial orientations, we will then obtain M k different features at level k in the hierarchy. To keep the complexity down at higher levels, we will for k ≥ K in a corresponding way as done by Hadji and Wildes [80] introduce a pooling stage over orientations…”

“…Inspired by the way the SURF descriptor [54] accumulates mean values and mean absolute values of derivative responses and the way Bruno and Mallat [78] and Hadji and Wildes [80] compute mean values of their hierarchical feature representations, we will initially explore reducing the QuasiQuadNet to just the mean values over the image domain of the following 5 features…”

Provably Scale-Covariant Networks from Oriented Quasi Quadrature Measures in Cascade

“…If we make the assumption that a spatial texture should obey certain stationarity properties over image space, we may regard it as reasonable to construct texture descriptors by accumulating statistics of feature responses over the image domain, in terms of e.g mean values or histograms. Inspired by the way the SURF descriptor [27] accumulates mean values and mean absolute values of derivative responses and the way Bruno and Mallat [24] and Hadji and Wildes [26] compute mean values of their hierarchical feature representations, we will initially explore reducing the QuasiQuadNet to just the mean values over the image domain of the following 5 features…”