Knowledge transfer is widely held to be a primary mechanism that enables humans to quickly learn new complex concepts when given only small training sets. In this paper, we apply knowledge transfer to deep convolutional neural nets, which we argue are particularly well suited for knowledge transfer. Our initial results demonstrate that components of a trained deep convolutional neural net can constructively transfer information to another such net. Furthermore, this transfer is completed in such a way that one can envision creating a net that could learn new concepts throughout its lifetime.

No abstract

Historically, neural nets have learned new things at the cost of forgetting what they already know. This problem is known as 'catastrophic forgetting'. Here, we examine how training a neural net in accordance with latently learned [1] output encodings drastically reduces catastrophic forgetting. Previous approaches to dealing with catastrophic forgetting have tended either to add extra samples to new training sets, modify the training of hidden nodes or model the interaction between short term and long term memory.Our approach is unique in that it both uses transfer learning to mitigate catastrophic forgetting and focuses upon the output nodes of a neural network. This results in a technique that makes it easier rather than harder to learn new tasks while retaining existing knowledge; is architecture independent and trivial to implement on any existing net.Additionally, we examine the use of ternary output codes.Binary codes assign a value to each output bit that may be thought of as either affirmative or negative. Ternary codes allow for the possibility that not every output bit has a meaningful response to every given input. By not forcing each output bit to train for a specific response for each new class, we hope to lessen catastrophic forgetting.

No abstract

No abstract