Self-Organization of Spatio-Temporal Hierarchy via Learning of Dynamic Visual Image Patterns on Action Sequences

Abstract

It is well known that the visual cortex efficiently processes high-dimensional spatial information by using a hierarchical structure. Recently, computational models that were inspired by the spatial hierarchy of the visual cortex have shown remarkable performance in image recognition. Up to now, however, most biological and computational modeling studies have mainly focused on the spatial domain and do not discuss temporal domain processing of the visual cortex. Several studies on the visual cortex and other brain areas associated with motor control support that the brain also uses its hierarchical structure as a processing mechanism for temporal information. Based on the success of previous computational models using spatial hierarchy and temporal hierarchy observed in the brain, the current report introduces a novel neural network model for the recognition of dynamic visual image patterns based solely on the learning of exemplars. This model is characterized by the application of both spatial and temporal constraints on local neural activities, resulting in the self-organization of a spatio-temporal hierarchy necessary for the recognition of complex dynamic visual image patterns. The evaluation with the Weizmann dataset in recognition of a set of prototypical human movement patterns showed that the proposed model is significantly robust in recognizing dynamically occluded visual patterns compared to other baseline models. Furthermore, an evaluation test for the recognition of concatenated sequences of those prototypical movement patterns indicated that the model is endowed with a remarkable capability for the contextual recognition of long-range dynamic visual image patterns.

Neural and Computational Mechanisms of Action Processing: Interaction between Visual and Motor Representations

“…Such deep architectures are presently among the best performing solutions for action classification from real videos in computer vision. Also, recently, deep architectures including dynamical neurons that learn hierarchical spatio-temporal representations have been proposed (Jung et al, 2015). Many details of the existing deep architectures (applied filter kernels, training schemes, regularization by ''drop out,'' etc.)…”

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

“…This shows that the states developed by the system can be suitably mapped onto lower level visual modalities. Thus, further developed models may hierarchically process lower level visual information similar to Jung et al (2015), however, based on top-down predicted, higher level, and bodily grounded motion estimates.…”

Finding Your Way from the Bed to the Kitchen: Reenacting and Recombining Sensorimotor Episodes Learned from Human Demonstration

“…Over the last two decades, computational models, such as HAMMER (Demiris and Hayes, 2002;Demiris and Khadhouri, 2006;Demiris et al, 2014), MOSAIC (Wolpert and Kawato, 1998;Haruno et al, 2003), MTRNN (Yamashita and Tani, 2008), and MSTNN (Jung et al, 2015), have received significant attention. These models all rely on prediction and to some extent a pairing of forward and inverse models, and some have been used for internal simulation.…”

“…Two computational models that have been used in more complex settings are HAMMER (Demiris and Simmons, 2006;Demiris et al, 2014) and the recurrent neural network models by Tani and colleagues Yamashita and Tani, 2008;Jung et al, 2015).…”