Specialization in Hierarchical Learning Systems

Hihn, Heinke; Braun, Daniel A.

doi:10.1007/s11063-020-10351-3

Cited by 14 publications

(11 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Secondly, we propose to use information processing constraints on the gating layer and the classifier based on a theory of bounded rationality (Ortega et al, 2015 ). To this end, we follow the work of Hihn and Braun ( 2020b ), where they introduce and motivate such constraints and show their favorable effects on generalization in the meta-learning setting (Hihn and Braun, 2020a ). We show that these types of constraints enable efficient representation learning, which is the pretext task.…”

Section: Methodsmentioning

confidence: 99%

“…This term is not tractable as the data generating distribution p ( x ) is unknown. We approximate the true marginal by running an exponential running mean with window length τ (Hihn and Braun, 2020b ; Leibfried and Grau-Moya, 2020 ):…”

Section: Methodsmentioning

confidence: 99%

“…We will refer to q ( h ) as p ( h ) to keep notation consistent. There are different interpretations of this approach, e.g., learning with information constraints (Hihn et al, 2018 , 2019 ; Hihn and Braun, 2020b ), meta-learning (Hihn and Braun, 2020a ), tempered posteriors in variational Bayes applications (Aitchison, 2021 ), and learning disentangled representations (Higgins et al, 2016 ). The additional degree of adaptivity introduced allows to learn data dependent priors which can improve the quality of generated samples, as we will show empirically in section 4.3.…”

Section: Methodsmentioning

confidence: 99%

“…To further improve this coupling, we propose to impose information-processing constraints on the latent representation, the gating layer, and on the classifier. We argue that such constraints encourage the system to learn representations that discover regularities in the data by discarding all unnecessary information (Hihn and Braun, 2020b ). To this end, we formulate an information-theoretic coupling as a two stage hierarchical system (Genewein et al, 2015 ) with the following objective function:…”

Section: Methodsmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective

et al. 2021

Self Cite

View full text Add to dashboard Cite

Traditional pain assessment approaches ranging from self-reporting methods, to observational scales, rely on the ability of an individual to accurately assess and successfully report observed or experienced pain episodes. Automatic pain assessment tools are therefore more than desirable in cases where this specific ability is negatively affected by various psycho-physiological dispositions, as well as distinct physical traits such as in the case of professional athletes, who usually have a higher pain tolerance as regular individuals. Hence, several approaches have been proposed during the past decades for the implementation of an autonomous and effective pain assessment system. These approaches range from more conventional supervised and semi-supervised learning techniques applied on a set of carefully hand-designed feature representations, to deep neural networks applied on preprocessed signals. Some of the most prominent advantages of deep neural networks are the ability to automatically learn relevant features, as well as the inherent adaptability of trained deep neural networks to related inference tasks. Yet, some significant drawbacks such as requiring large amounts of data to train deep models and over-fitting remain. Both of these problems are especially relevant in pain intensity assessment, where labeled data is scarce and generalization is of utmost importance. In the following work we address these shortcomings by introducing several novel multi-modal deep learning approaches (characterized by specific supervised, as well as self-supervised learning techniques) for the assessment of pain intensity based on measurable bio-physiological data. While the proposed supervised deep learning approach is able to attain state-of-the-art inference performances, our self-supervised approach is able to significantly improve the data efficiency of the proposed architecture by automatically generating physiological data and simultaneously performing a fine-tuning of the architecture, which has been previously trained on a significantly smaller amount of data.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…During each forward pass, the gating policy selects the top-k experts, such that the output of the layer is a weighted sum of k experts. To enable continual learning, we leverage an information-theoretic formulation of specialization in multi-expert systems [17] that emerged from an information-theoretic formulation of bounded rationality [34,10]. Specialized experts focus on a particular sub-region of inputs, enabling them to adapt quickly to new data.…”

Section: Introductionmentioning

confidence: 99%

Mixture-of-Variational-Experts for Continual Learning

Hihn¹,

Braun²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

One significant shortcoming of machine learning is the poor ability of models to solve new problems quicker and without forgetting acquired knowledge. To better understand this issue, continual learning has emerged to systematically investigate learning protocols where the model sequentially observes samples generated by a series of tasks. First, we propose an optimality principle that facilitates a trade-off between learning and forgetting. We derive this principle from an information-theoretic formulation of bounded rationality and show its connections to other continual learning methods. Second, based on this principle, we propose a neural network layer for continual learning, called Mixture-of-Variational-Experts (MoVE), that alleviates forgetting while enabling the beneficial transfer of knowledge to new tasks. Our experiments on variants of the MNIST and CI-FAR10 datasets demonstrate the competitive performance of MoVE layers when compared to stateof-the-art approaches.

show abstract