Neurogenesis deep learning: Extending deep networks to accommodate new classes

Abstract

Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing -data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

Lifelong Learning Starting from Zero

“…Other incremental methods exist, based e.g. on incremental training of an auto-encoder; new neurons are added in response to a high rate of failure with the new data [31] or based on reconstruction error [5]. AdaNet [3] gradually extends its network by evaluation and selection among candidate sub-networks.…”

Probabilistic Program Neurogenesis

“…There are a variety of methodologies for tackling the continual learning problem [Parisi et al, 2019]. Existing approaches can be roughly divided into three categories: those that focus on consolidating synapses in a neural network that are critical to retain previous knowledge [Kirkpatrick et al, 2017;Zenke et al, 2017;Aljundi et al, 2018;; those that employ either an explicit memory buffer or a generative model to be able to interleave the learning of new knowledge with that based on stored or generated data for the old tasks [McClelland et al, 1995;Ans et al, 2004;Atkinson et al, 2018;; and those that dynamically change the structure of the neural network to accommodate new knowledge [Rusu et al, 2016;Draelos et al, 2017;Yoon et al, 2017]. Here, we address the challenge of continual learning by intelligently adding new neurons to an existing neural network (neurogenesis) so that it learns to solve a new task without forgetting how to solve previously learned tasks.…”

“…In contrast, our approach learns where and how many neurons to add. Draelos et al [2017] proposed the neurogenesis deep learning (NDL) model to incrementally train an autoencoder on new MNIST digits by adding new neurons at each layer and employing a pseudo-rehearsal process called "intrinsic replay" for preserve the performance on the old MNIST digits. Yoon et al [2017] proposed a dynamically expanding network (DEN) that selectively adapts network weights and also expand network structure at each layer as needed using group sparse regularization in an online manner to facilitate the sequential learning of tasks.…”

Um Método para Avaliação Automática da Qualidade de Recursos Educacionais Abertos usando Deep Learning

“…Mas, esse significado mudou ao longo do tempo [Bengio et al 2009, LeCun et al 2015, Draelos et al 2017. Em 2012, 10 camadas já eram suficientes para considerar uma rede neural como profunda.…”

“…Em 2012, 10 camadas já eram suficientes para considerar uma rede neural como profunda. Contudo, em 2017é mais comum considerar uma rede neural profunda quando esta possui mais de 100 camadas [Draelos et al 2017]. Deep Learning foi aplicado em diversos domínios como imagem, texto, vídeo, pronúncia e visão, melhorando significativamente os melhores resultados alcançados em dezenas de anos.…”