Comparison between Frame-Constrained Fix-Pixel-Value and Frame-Free Spiking-Dynamic-Pixel ConvNets for Visual Processing

Farabet, Clément; Paz, Rafael; Pérez-Carrasco, Jose Antonio; Zamarreno-Ramos, C.; Linares-Barranco, A.; LeCun, Yann; Culurciello, Eugenio; Serrano-Gotarredona, Teresa; Linares-Barranco, B.

doi:10.3389/fnins.2012.00032

Cited by 63 publications

(46 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, as we have shown, spiking DBNs can process data with very low latency, without having to wait for a full frame of data, which can be further improved if individual units of the DBN compute in parallel, rather than updating each unit in sequence. This advantage has been recognized for many years for feed-forward convolutional networks, in which almost all operations can be efficiently parallelized, and has led to the development of custom digital hardware solutions and spike-based convolution chips (Camuñas Mesa et al, 2010; Farabet et al, 2012), which through the use of the Address Event Representation (AER) protocol, can also directly process events coming from event-based dynamic vision sensors (Lichtsteiner et al, 2008). For such architectures (Pérez-Carrasco et al, 2013) have recently developed a similar mapping methodology between frame-based and event-driven networks that translates the weights and other parameters of a fully trained frame-based feed-forward network into the event-based domain, and then optimizes them with simulated annealing.…”

Section: Discussionmentioning

confidence: 99%

“…In comparison, this offers increased flexibility to change neuronal parameters after training, whereas our method uses the accurate Siegert-approximation of spike rates already during the training of a bi-directional network, and does not require an additional optimization phase. The advantages of spike-based versus digital frame-based visual processing in terms of processing speed and scalability have been compared in Farabet et al (2012), where it was also suggested that spike-based systems are more suitable for systems that employ both feed-forward and feed-back processing.…”

Section: Discussionmentioning

confidence: 99%

“…This scheme would allow the system to remain silent, consuming little power, in potentially long silent periods, and still allow fast recognition when bursts of input activity arrive, a scenario that is realistic for natural organisms. These advantages have been recently realized for event-based convolutional networks using convolution chips (Camuñas Mesa et al, 2010; Farabet et al, 2012), but a principled way of building DBNs models out of spiking neurons, in which both feed-forward and feed-back processing are implemented has been lacking.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Real-time classification and sensor fusion with a spiking deep belief network

et al. 2013

View full text Add to dashboard Cite

Deep Belief Networks (DBNs) have recently shown impressive performance on a broad range of classification problems. Their generative properties allow better understanding of the performance, and provide a simpler solution for sensor fusion tasks. However, because of their inherent need for feedback and parallel update of large numbers of units, DBNs are expensive to implement on serial computers. This paper proposes a method based on the Siegert approximation for Integrate-and-Fire neurons to map an offline-trained DBN onto an efficient event-driven spiking neural network suitable for hardware implementation. The method is demonstrated in simulation and by a real-time implementation of a 3-layer network with 2694 neurons used for visual classification of MNIST handwritten digits with input from a 128 × 128 Dynamic Vision Sensor (DVS) silicon retina, and sensory-fusion using additional input from a 64-channel AER-EAR silicon cochlea. The system is implemented through the open-source software in the jAER project and runs in real-time on a laptop computer. It is demonstrated that the system can recognize digits in the presence of distractions, noise, scaling, translation and rotation, and that the degradation of recognition performance by using an event-based approach is less than 1%. Recognition is achieved in an average of 5.8 ms after the onset of the presentation of a digit. By cue integration from both silicon retina and cochlea outputs we show that the system can be biased to select the correct digit from otherwise ambiguous input.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Real-time classification and sensor fusion with a spiking deep belief network

et al. 2013

View full text Add to dashboard Cite

show abstract

“…The combination of spiking deep networks together with event-based sensors has been considered in previous studies, for a example, through a spiking CNN receiving DVS spikes [5], [6]. Spiking deep networks have also been implemented in hardware, for example, a spiking DBN was implemented on a hardware platform and interfaced to a DVS [7], [8].…”

Section: Introductionmentioning

confidence: 99%

Effective sensor fusion with event-based sensors and deep network architectures

Neil

Liu

2016

2016 IEEE International Symposium on Circuits and Systems (ISCAS)

View full text Add to dashboard Cite

The use of spiking neuromorphic sensors with state-of-art deep networks is currently an active area of research. Still relatively unexplored are the pre-processing steps needed to transform spikes from these sensors and the types of network architectures that can produce high-accuracy performance using these sensors. This paper discusses several methods for preprocessing the spiking data from these sensors for use with various deep network architectures. The outputs of these preprocessing methods are evaluated using different networks including a deep fusion network composed of Convolutional Neural Networks and Recurrent Neural Networks, to jointly solve a recognition task using the MNIST (visual) and TIDIGITS (audio) benchmark datasets. With only 1000 visual input spikes from a spiking hardware retina, the classification accuracy of 64.5% achieved by a particular trained fusion network increases to 98.31% when combined with inputs from a spiking hardware cochlea. Abstract-The use of spiking neuromorphic sensors with stateof-art deep networks is currently an active area of research. Still relatively unexplored are the pre-processing steps needed to transform spikes from these sensors and the types of network architectures that can produce high-accuracy performance using these sensors. This paper discusses several methods for preprocessing the spiking data from these sensors for use with various deep network architectures. The outputs of these preprocessing methods are evaluated using different networks including a deep fusion network composed of Convolutional Neural Networks and Recurrent Neural Networks, to jointly solve a recognition task using the MNIST (visual) and TIDIGITS (audio) benchmark datasets. With only 1000 visual input spikes from a spiking hardware retina, the classification accuracy of 64.5% achieved by a particular trained fusion network increases to 98.31% when combined with inputs from a spiking hardware cochlea.

show abstract

“…In one case, the convolutional neural network (CNN) units were translated into biologically inspired spiking units with leaks and refractory periods. [4,5] SNNs can output results even after the first spike is produced, [6] whereas in ANNs the result is only available when all the layers have been totally processed. [6] This scheme is based on the principle of rescaling the weights to avoid approximation errors in SNNs due to either excessive or too little firing of the neurons.…”

mentioning

confidence: 99%

Artificial Neural Network (ANN) to Spiking Neural Network (SNN) Converters Based on Diffusive Memristors

Midya

Wang

Asapu

et al. 2019

Adv Elect Materials

109

View full text Add to dashboard Cite

also been shown to be accurate, fast, and efficient when implementing object recognition or detection on neuromorphic hardware platforms. [8] It is then natural to look for a middle path by consolidating the advantages of these two types of networks in a single computing system. The key to bridging the gap between continuous valued ANNs and neuromorphic spiking networks is the necessity to develop SNNs that can match the error rates of their continuous valued ANNs. There have been a few efforts toward this direction such as training SNNs using backpropagation, [9] implementing SNN classification layers using stochastic gradient descent [10] or modifying the transfer function of ANN during training so that the network parameters can be mapped to the SNN. [4,11] Although these results are promising, these methods are not sufficiently efficient to train a spiking architecture of the size of VGG-16 yet. A seemingly easier approach would be to take the outputs of a pretrained ANN and then map them to an equivalently accurate SNN. There have been a few efforts in the field of ANN-SNN conversion. In one case, the convolutional neural network (CNN) units were translated into biologically inspired spiking units with leaks and refractory periods. [12] In another report, nearly lossless conversion of ANNs for the Modified National Institute of Standards and Technology (MNIST) [13] classification task was achieved by using a weight normalization scheme. [6] This scheme is based on the principle of rescaling the weights to avoid approximation errors in SNNs due to either excessive or too little firing of the neurons. Researchers in IBM demonstrated an approach that optimized CNNs for the TrueNorth platform, which has binary weights and restricted connectivity. [14] In another study along similar lines, a conversion method was developed which involved spiking neurons that adapted their firing threshold to reduce the number of spikes needed to encode information. [15] However, such studies were all limited to conventional complimentary metal oxide semiconductor (CMOS)-based hardware and not much effort has been invested in taking advantage of this relatively important new paradigm (ANN-SNN conversion) and the inherent advantages possibly offered by emerging nanoscale devices, such as memristors.The ORCID identification number(s) for the author(s) of this article can be found under https://doi.org/10.1002/aelm.201900060. Artificial Neural NetworksThe bar for state of the art classification error rates has been pushed to new levels by GoogLeNet [1] and VGG-16 [2] for computer vision benchmarks such as ImageNet. [3] Artificial neural networks (ANNs) have shown remarkable performance in performing tasks of practical importance such as image recognition, edge detection, decision making, sequence recognition, and playing Go games, just to name a few. On the other hand, spiking neural networks (SNNs) are very effective at reducing the latency and computational load of deep neural networks. [4,5] SNNs can output results even after the ...

show abstract

Comparison between Frame-Constrained Fix-Pixel-Value and Frame-Free Spiking-Dynamic-Pixel ConvNets for Visual Processing

Cited by 63 publications

References 48 publications

Real-time classification and sensor fusion with a spiking deep belief network

Real-time classification and sensor fusion with a spiking deep belief network

Effective sensor fusion with event-based sensors and deep network architectures

Artificial Neural Network (ANN) to Spiking Neural Network (SNN) Converters Based on Diffusive Memristors

Contact Info

Product

Resources

About