Learning entropy for novelty detection a cognitive approach for adaptive filters

Bukovský, Ivo; Oswald, Cyril; Cejnek, Matouš; Beneš, Petr

doi:10.1109/sspd.2014.6943329

Cited by 15 publications

(10 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concept of learning entropy is based on the evaluation of unusual weight updates as the unusual learning pattern can indicate novelty in training data; i.e., the new information that new samples of data carry in respect to what the NN contemporary has learned already [34]. This methodology to evaluate the learning entropy through the unusually large weight updates was recently introduced [34] and then reviewed with some simplifications recently in [35,36,38]. The first important parameters here are as follows:…”

Section: The Multiscale Approachmentioning

confidence: 99%

“…The motivation of this brief paper is to discuss and extend the recently introduced concept of LE [34,35] in the sense of a (machine) learning-based information measure as a founding concept of the cognitive novelty detection based on quantification of unusual learning efforts of learning systems. Novelty detection via LE is based on real-time learning of systems after they had been pretrained on an initial pretraining data set.…”

Section: Introductionmentioning

confidence: 99%

“…Section 2 first discusses a loose parallel between entropy in the sense of thermodynamics and importantly, the distinction from the concept of Shannon's information theory, with the proposal of the concept of (machine) learning-based information measure. Secondly, the original multiscale LE algorithm [34,35] based on unusually large learning efforts is reviewed and followed by its approximate version. Thirdly, it provides an alternative (more direct) formula for practical computations.…”

Section: Introductionmentioning

confidence: 99%

“…In fact, the learning entropy E in (18) is considered to be the first-order learning entropy because the detection function (13) is calculated with the first difference (≈ first-order derivative) of weights (as it results from (11)). It appeared useful to practically enhance LE computation with higher-order differences of weight updates that contribute to more reliable novelty detection as the higher order weight difference terms indicates useful noise filtering [34][35][36]38]. To compute the LE of various orders, the corresponding weight differences can be used in formulas (12)- (14) as in Table 1.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Learning Entropy as a Learning-Based Information Concept

Bukovský

Kinsner

Homma

2019

Entropy

Self Cite

View full text Add to dashboard Cite

Recently, a novel concept of a non-probabilistic novelty detection measure, based on a multi-scale quantification of unusually large learning efforts of machine learning systems, was introduced as learning entropy (LE). The key finding with LE is that the learning effort of learning systems is quantifiable as a novelty measure for each individually observed data point of otherwise complex dynamic systems, while the model accuracy is not a necessary requirement for novelty detection. This brief paper extends the explanation of LE from the point of an informatics approach towards a cognitive (learning-based) information measure emphasizing the distinction from Shannon’s concept of probabilistic information. Fundamental derivations of learning entropy and of its practical estimations are recalled and further extended. The potentials, limitations, and, thus, the current challenges of LE are discussed.

show abstract

Section: The Multiscale Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Learning Entropy as a Learning-Based Information Concept

Bukovský

Kinsner

Homma

2019

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…Tab. 1: Extensions of adaptive learning rates for HONU [45]; the adaptive learning rate then still can be used in stability conditions (29) Based on Normalized Least Mean Squares (or Normalized GD)…”

Section: Multiple Learning Ratesmentioning

confidence: 99%

An Approach to Stable Gradient-Descent Adaptation of Higher Order Neural Units

Bukovský

Homma

2017

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

Abstract-Stability evaluation of a weight-update system of higher-order neural units (HONUs) with polynomial aggregation of neural inputs (also known as classes of polynomial neural networks) for adaptation of both feedforward and recurrent HONUs by a gradient descent method is introduced. An essential core of the approach is based on spectral radius of a weight-update system, and it allows stability monitoring and its maintenance at every adaptation step individually. Assuring stability of the weight-update system (at every single adaptation step) naturally results in adaptation stability of the whole neural architecture that adapts to target data. As an aside, the used approach highlights the fact that the weight optimization of HONU is a linear problem, so the proposed approach can be generally extended to any neural architecture that is linear in its adaptable parameters.

show abstract

3D Lung Tumor Segmentation System Using Adaptive Structural Deep Belief Network

Kamada,

Ichimura

2024

Intelligent Systems Reference Library

View full text Add to dashboard Cite

Learning entropy for novelty detection a cognitive approach for adaptive filters

Cited by 15 publications

References 14 publications

Learning Entropy as a Learning-Based Information Concept

Learning Entropy as a Learning-Based Information Concept

An Approach to Stable Gradient-Descent Adaptation of Higher Order Neural Units

3D Lung Tumor Segmentation System Using Adaptive Structural Deep Belief Network

Contact Info

Product

Resources

About