Direct Feedback Alignment With Sparse Connections for Local Learning

Abstract

Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. We evaluate our algorithm using standard datasets, including ImageNet, to address the concern of scaling to challenging problems. Our results show orders of magnitude improvement in data movement and 2× improvement in multiply-and-accumulate operations over backpropagation. Like previous work, we observe that any variant of feedback alignment suffers significant losses in classification accuracy on deep convolutional neural networks. By transferring trained convolutional layers and training the fully connected layers using direct feedback alignment, we demonstrate that direct feedback alignment can obtain results competitive with backpropagation. Furthermore, we observe that using an extremely sparse feedback matrix, rather than a dense one, results in a small accuracy drop while yielding hardware advantages. All the code and results are available under https://github.com/bcrafton/ssdfa .

Learning to solve the credit assignment problem

“…First, training of the forward and backward weights may be performed separately, and hence the forward and backward pass of backpropagation may be performed asynchronously. Thus the method may be applied on distributed systems in which synchronization is difficult or time consuming [10,9], including some integrated circuits [1]. Second, by relying on random perturbations to measure gradients, the method does not rely on the environment to provide gradients.…”

HebbNet: A Simplified Hebbian Learning Framework to do Biologically Plausible Learning

“…Work has also been done to use it as an alternative to backpropagation [13,14,15,16,17], however, the results have not been so strong. Hebbian learning differs from other forms of biologically plausible learning such as feedback alignment [18,19,20], target propagation [21,22] and others [23,24], in that it is completely unsupervised and no information or feedback in any form is passed to it.…”

Stochastic dendrites enable online learning in mixed-signal neuromorphic processing systems

“…However, endowing these systems with online learning abilities remains an open challenge. Since providing a top-down error signal has been very successful in deep learning [7], [8], some neuromorphic implementations have recently focused on using errors for online learning [9], [10]. Moreover, many neuroscientific studies have recently focused on error-based learning in the brain [11], [12].…”