HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition

Abstract

This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.

Invariant feature extraction from event based stimuli

“…We discard the spatial and temporal information of the output events, and bin them based on the event type to create a histogram. The normalized histogram is used as the feature for object recognition [11]. Such histograms are calculated for all the instances in the training and testing data.…”

“…The "Faces" data set introduced in [11] poses a more difficult classification task. This dataset contains 24 samples each from 7 subjects.…”

“…The value ( , , ) ap  represents the weight attributed to inputs from the spatial location ( , )  and event type p . For each of the 11 w w  spatial locations we obtain the weight with the largest absolute value for all the input event types. This indicates the type of first layer event that would elicit the maximum response from this subspace at that particular spatial location.…”

“…This indicates the type of first layer event that would elicit the maximum response from this subspace at that particular spatial location. We plot the selected first layer subspaces for all the corresponding 11 w w  spatial locations and scale the contrast of the individual sub-images according to the absolute value of the associated weight. In Figure 4.…”

“…Feature extractors applicable for such stimuli should be invariant to the fast changes in the input signal, and extract the underlying static object attributes. In [11] a hierarchical feature extraction method based on learning timesurface prototypes has been introduced, but the issue of invariance was not addressed explicitly.…”