Backpropagation Applied to Handwritten Zip Code Recognition

LeCun, Yann; Boser, Bernhard E.; Denker, John S.; Henderson, D.; Howard, Richard; Hubbard, W.; Jackel, L. D.

doi:10.1162/neco.1989.1.4.541

Cited by 10,006 publications

(5,219 citation statements)

References 5 publications

Supporting

Mentioning

5,112

Contrasting

Unclassified

101

Order By: Relevance

“…19-5.23). This work also introduced the MNIST data set of handwritten digits (LeCun et al, 1989), which over time has become perhaps the most famous benchmark of Machine Learning. CNNs helped to achieve good performance on MNIST (LeCun et al, 1990a) (CAP depth 5) and on fingerprint recognition (Baldi and Chauvin, 1993); similar CNNs were used commercially in the 1990s.…”

Section: : Bp For Convolutional Nns (Cnns Sec 54)mentioning

confidence: 99%

“…5.11) CNNs (Fukushima, 1979;LeCun et al, 1989) Multi-Column GPU-MPCNNs (Ciresan et al, 2011b) are committees (Breiman, 1996;Schapire, 1990;Wolpert, 1992;Hashem and Schmeiser, 1992;Ueda, 2000;Dietterich, 2000a) of GPUMPCNNs with simple democratic output averaging. Several MPCNNs see the same input; their output vectors are used to assign probabilities to the various possible classes.…”

Section: : Mpcnns On Gpu Achieve Superhuman Vision Performancementioning

confidence: 99%

See 1 more Smart Citation

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

Section: : Bp For Convolutional Nns (Cnns Sec 54)mentioning

confidence: 99%

Section: : Mpcnns On Gpu Achieve Superhuman Vision Performancementioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

show abstract

“…We use deep learning techniques [13][14][15][16] to infer model parameters and to optimize algorithm settings. Our training pipeline (Fig.…”

Section: Training Deepbind and Scoring Sequencesmentioning

confidence: 99%

Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning

et al. 2015

View full text Add to dashboard Cite

Knowing the sequence specificities of DNA-and RNA-binding proteins is essential for developing models of the regulatory processes in biological systems and for identifying causal disease variants. Here we show that sequence specificities can be ascertained from experimental data with 'deep learning' techniques, which offer a scalable, flexible and unified computational approach for pattern discovery. Using a diverse array of experimental data and evaluation metrics, we find that deep learning outperforms other state-of-the-art methods, even when training on in vitro data and testing on in vivo data. We call this approach DeepBind and have built a stand-alone software tool that is fully automatic and handles millions of sequences per experiment. Specificities determined by DeepBind are readily visualized as a weighted ensemble of position weight matrices or as a 'mutation map' that indicates how variations affect binding within a specific sequence.DNA-and RNA-binding proteins play a central role in gene regulation, including transcription and alternative splicing. The sequence specificities of a protein are most commonly characterized using position weight matrices 1 (PWMs), which are easy to interpret and can be scanned over a genomic sequence to detect potential binding sites. However, growing evidence indicates that sequence specificities can be more accurately captured by more complex techniques 2-5 . Recently, 'deep learning' has achieved record-breaking performance in a variety of information technology applications 6,7 . We adapted deep learning methods to the task of predicting sequence specificities and found that they compete favorably with the state of the art. Our approach, called DeepBind, is based on deep convolutional neural networks and can discover new patterns even when the locations of patterns within sequences are unknown-a task for which traditional neural networks require an exorbitant amount of training data.There are several challenging aspects in learning models of sequence specificity using modern high-throughput technologies. First, the data come in qualitatively different forms. Protein binding microarrays (PBMs) 8 and RNAcompete assays 9 provide a specificity coefficient for each probe sequence, whereas chromatin immunoprecipitation (ChIP)-seq 10 provides a ranked list of putatively bound sequences of varying length, and HT-SELEX 11 generates a set of very high affinity sequences. Second, the quantity of data is large. A typical high-throughput experiment measures between 10,000 and 100,000 sequences, and it is computationally demanding to incorporate them all. Third, each data acquisition technology has its own artifacts, biases and limitations, and we must discover the pertinent specificities despite these unwanted effects. For example, ChIP-seq reads often localize to "hyper-ChIPable" regions of the genome near highly expressed genes 12 .DeepBind (Fig. 1) addresses the above challenges. (i) It can be applied to both microarray and sequencing data; (ii) it can learn from millions of...

show abstract

“…In the first layer, these filters are able to detect for instance edges and very simple shapes, but composing a hierarchy of these filters allows for great compositional power to express complex features and is an important reason DNNs have proven to be so successful. As determining these filters by hand is practically impossible DNNs are trained by backpropagation (LeCun et al, 1989), a standard machine learning optimization method based on gradient descent. Given a cost function that determines for an input and an expected output a single error value, backpropagation allows to assign a credit to each single unit in the network to specify how much it contributed to the error.…”

Section: Box 1 Deep Neural Networkmentioning

confidence: 99%

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Scholte

Losch

Ramakrishnan

et al. 2018

Cortex

View full text Add to dashboard Cite

Deep learning Cost functions RepresentationsVisual processing a b s t r a c t Vision research has been shaped by the seminal insight that we can understand the higher-tier visual cortex from the perspective of multiple functional pathways with different goals. In this paper, we try to give a computational account of the functional organization of this system by reasoning from the perspective of multi-task deep neural networks. Machine learning has shown that tasks become easier to solve when they are decomposed into subtasks with their own cost function. We hypothesize that the visual system optimizes multiple cost functions of unrelated tasks and this causes the emergence of a ventral pathway dedicated to vision for perception, and a dorsal pathway dedicated to vision for action.To evaluate the functional organization in multi-task deep neural networks, we propose a method that measures the contribution of a unit towards each task, applying it to two networks that have been trained on either two related or two unrelated tasks, using an identical stimulus set. Results show that the network trained on the unrelated tasks shows a decreasing degree of feature representation sharing towards higher-tier layers while the network trained on related tasks uniformly shows high degree of sharing.We conjecture that the method we propose can be used to analyze the anatomical and functional organization of the visual system and beyond. We predict that the degree to which tasks are related is a good descriptor of the degree to which they share downstream cortical-units.

show abstract

Backpropagation Applied to Handwritten Zip Code Recognition

Cited by 10,006 publications

References 5 publications

Deep learning in neural networks: An overview

Deep learning in neural networks: An overview

Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Contact Info

Product

Resources

About