Modular deep belief networks that do not forget

Pape, Leo; Gomez, Faustino; Ring, Mark; Schmidhuber, Jürgen

doi:10.1109/ijcnn.2011.6033359

Cited by 12 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Introducing modularization to prevent forgetting dates back to [42] on the training of deep belief networks (DBN) [27]. Recently, [54] suggests a modular solution by identifying the trained modules (groups of neurons) to be re-used and extending the network with new modules for each new task.…”

Section: Regularization-based Continual Learning and Network Modulari...mentioning

confidence: 99%

Learning to Transfer with von Neumann Conditional Divergence

Shaker¹,

Yu²,

Oñoro-Rubio³

2021

Preprint

View full text Add to dashboard Cite

The similarity of feature representations plays a pivotal role in the success of domain adaptation and generalization. Feature similarity includes both the invariance of marginal distributions and the closeness of conditional distributions given the desired response y (e.g., class labels). Unfortunately, traditional methods always learn such features without fully taking into consideration the information in y, which in turn may lead to a mismatch of the conditional distributions or the mix-up of discriminative structures underlying data distributions. In this work, we introduce the recently proposed von Neumann conditional divergence to improve the transferability across multiple domains. We show that this new divergence is differentiable and eligible to easily quantify the functional dependence between features and y. Given multiple source tasks, we integrate this divergence to capture discriminative information in y and design novel learning objectives assuming those source tasks are observed either simultaneously or sequentially. In both scenarios, we obtain favorable performance against state-of-the-art methods in terms of smaller generalization error on new tasks and less catastrophic forgetting on source tasks (in the sequential setup).

show abstract

Section: Regularization-based Continual Learning and Network Modulari...mentioning

confidence: 99%

Learning to Transfer with von Neumann Conditional Divergence

Shaker¹,

Yu²,

Oñoro-Rubio³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…One variation of DBN, called modular DBN (M-DBN), trains different parts of the network separately, while adjusting the learning rate as training progresses (Pape et al 2011), as opposed to using one training set for the whole network. This allows M-DBN to avoid forgetting features learned early in training, a weakness of DBN that can hinder its performance in online learning applications in which the data distribution changes dynamically over time.…”

Section: Deep Network Similar To Dbnmentioning

confidence: 99%

Efficient Learning Machines

Awad¹,

Khanna²

2015

665

157

View full text Add to dashboard Cite

Machine LearningNature is a self-made machine, more perfectly automated than any automated machine. To create something in the image of nature is to create a machine, and it was by learning the inner working of nature that man became a builder of machines.-Eric Hoffer, Reflections on the Human Condition Machine learning (ML) is a branch of artificial intelligence that systematically applies algorithms to synthesize the underlying relationships among data and information. For example, ML systems can be trained on automatic speech recognition systems (such as iPhone's Siri) to convert acoustic information in a sequence of speech data into semantic structure expressed in the form of a string of words.ML is already finding widespread uses in web search, ad placement, credit scoring, stock market prediction, gene sequence analysis, behavior analysis, smart coupons, drug development, weather forecasting, big data analytics, and many more applications. ML will play a decisive role in the development of a host of user-centric innovations.ML owes its burgeoning adoption to its ability to characterize underlying relationships within large arrays of data in ways that solve problems in big data analytics, behavioral pattern recognition, and information evolution. ML systems can moreover be trained to categorize the changing conditions of a process so as to model variations in operating behavior. As bodies of knowledge evolve under the influence of new ideas and technologies, ML systems can identify disruptions to the existing models and redesign and retrain themselves to adapt to and coevolve with the new knowledge.The computational characteristic of ML is to generalize the training experience (or examples) and output a hypothesis that estimates the target function. The generalization attribute of ML allows the system to perform well on unseen data instances by accurately predicting the future data. Unlike other optimization problems, ML does not have a well-defined function that can be optimized. Instead, training errors serve as a catalyst to test learning errors. The process of generalization requires classifiers that input discrete or continuous feature vectors and output a class.The goal of ML is to predict future events or scenarios that are unknown to the computer. In 1959, Arthur Samuel described ML as the "field of study that gives computers the ability to learn without being explicitly programmed" (Samuel 1959). He concluded that programming computers to learn from experience should eventually eliminate the need for much of this detailed programming effort. According to Tom M. Mitchell's definition of ML: "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E." Alan Turing's seminal paper (Turing 1950) introduced a benchmark standard for demonstrating machine intelligence, such that a machine has to be intelligent and responsive in a manner that cannot be differentiated fro...

show abstract

“…However, some works have started addressing this issue in the context of deep networks. First, [67] proposes an organization in which new data are affected to the training of different subnetworks according to their current capacities. Another interesting approach was proposed by [8]: since deep networks are trained to generate their input data, these generated data can be used to extend the dataset.…”

Section: Perspectives and Future Workmentioning

confidence: 99%

Deep unsupervised network for multimodal perception, representation and classification

Droniou

Ivaldi

Sigaud

2015

Robotics and Autonomous Systems

View full text Add to dashboard Cite

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.

show abstract

Modular deep belief networks that do not forget

Cited by 12 publications

References 19 publications

Learning to Transfer with von Neumann Conditional Divergence

Learning to Transfer with von Neumann Conditional Divergence

Efficient Learning Machines

Deep unsupervised network for multimodal perception, representation and classification

Contact Info

Product

Resources

About