Bridge Networks

Abstract

Despite rapid progress, current deep learning methods face a number of critical challenges. These include high energy consumption, catastrophic forgetting, dependance on global losses, and an inability to reason symbolically. By combining concepts from information theory and vector-symbolic architectures, we propose and implement a novel information processing architecture, the 'Bridge network.' We show this architecture provides unique advantages which can address the problem of global losses and catastrophic forgetting. Furthermore, we argue that it provides a further basis for increasing energy efficiency of execution and the ability to reason symbolically.

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

“…Yet another promising avenue is make the processes of classification and reconstruction (i.e., generation) of raw sensory data simultaneously. One particular realization of this idea, called "bridge networks, " was recently presented in [304]. Finally, it is worth mentioning that a neural network does not necessarily need to produce HVs, but it can benefit from the HDC/VSA operations by improving its retrieval performance through superimposing multiple permuted versions of an output vector, as demonstrated in [56].…”

“…Another approach would be to discover general principles for combining neural networks and HDC/VSA, but currently there are only few such efforts [36,37,143,144,304,431] (see also Section 4.2.6). For example, the Tensor Product Representations operations in [36], and the HDC/VSA operations in [144], are introduced into the neural network machinery.…”

Training Spiking Networks via Natural Evolution Strategies

“…Furthermore, the correspondence of phase-based representations with what is known as a 'vector-symbolic system' allows vectors of activations to be combined and manipulated algebraically. This ability to describe, combine, and manipulate phase-based information at a high level enables the construction of deep, complex networks which operate similarly to residual and attention-based architectures [13]. The utilization of phase-based information at all layers in these networks yields a flexible system which can compute by exchanging packets of precisely-timed binary spikes.…”

“…In my previous work, an additional 'bias' input was applied to phasor neurons [13]. This change shifted the distribution of activations -each representing an angle -from uniform across the unit circle to normally-distributed around zero (Figure 2a).…”

“…This change allowed for the creation of residual blocks of phasor neurons -layers which approximate the identity function when initialized. This enabled very deep phasor networks which were successfully trained with backpropagation [3], [13]. This change also led to different behavior in the spiking domain.…”