The AIC Criterion and Symmetrizing the Kullback–Leibler Divergence

Seghouane, Abd‐Krim; Амари, Шун-ичи

doi:10.1109/tnn.2006.882813

Cited by 134 publications

(56 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The pertinent variables of the regression model are determined through Stepwise procedure (Bendel and Afifi 1977) which eliminates variables that are not statistically significant in the model from the Akaike Information Criterion (this criterion gives the information lost for each candidate model and the pertinent model is the one with low AIC) (Seghouane and Amari 2007). In addition, the significance of the correlation between the selected pertinent variables is assessed with the Pearson test of correlation (Millot 2009).…”

Section: Elaboration Of the Multiple Linear Regression Of The Annual mentioning

confidence: 99%

Changes in rainfall regime over Burkina Faso under the climate change conditions simulated by 5 regional climate models

et al. 2013

View full text Add to dashboard Cite

Sahelian rainfall has recorded a high variability during the last century with a significant decrease (more than 20 %) in the annual rainfall amount since 1970. Using a linear regression model, the fluctuations of the annual rainfall from the observations over Burkina Faso during 1961-2009 period are described through the changes in the characteristics of the rainy season. The methodology is then applied to simulated rainfall data produced by five regional climate models under A1B scenario over two periods: 1971-2000 as reference period and 2021-2050 as projection period. As found with other climate models, the projected change in annual rainfall for West Africa is very uncertain. However, the present study shows that some features of the impact of climate change on rainfall regime in the region are robust. The number of the low rainfall events (0.1-5 mm/d) is projected to decrease by 3 % and the number of strong rainfall events ([50 mm/d) is expected to increase by 15 % on average. In addition, the rainy season onset is projected by all models to be delayed by one week on average and a consensus exists on the lengthening of the dry spells at about 20 %. Furthermore, the simulated relationship between changed annual rainfall amounts and the number of rain days or their intensity varies strongly from one model to another and some changes do not correspond to what is observed for the rainfall variability over the last 50 years.

show abstract

Section: Elaboration Of the Multiple Linear Regression Of The Annual mentioning

confidence: 99%

Changes in rainfall regime over Burkina Faso under the climate change conditions simulated by 5 regional climate models

et al. 2013

View full text Add to dashboard Cite

show abstract

“…In the experiments we assumed a fixed number of hidden units. However, there is plenty of works on how to choose proper structure of a model using the model selection with different complexity penalty terms [2], naming only a few, Akaike's information theoretic criterion [1,32] or Rissanen's minimum description length [29], Last but not least, in this work we considered a probabilistic deep model. However, there is a vast of deterministic deep models based on auto-encoders.…”

Section: Resultsmentioning

confidence: 99%

Learning Informative Features from Restricted Boltzmann Machines

Tomczak

2015

Neural Process Lett

View full text Add to dashboard Cite

In recent years deep learning paradigm achieved important empirical success in a number of practical applications such as object recognition, speech recognition and natural language processing. A lot of effort has been put on understanding theoretical aspects of this success, however, still there is no common view on how deep architectures should be trained and thus many open questions remain. One hypothesis focuses on formulating good criterion (prior) that may help to learn a set of features capable of disentangling hidden factors. Following this line of thinking, in this paper, we propose to add a penalty (regularization) term to the log-likelihood function that enforces hidden units to maximize entropy and to be pairwise uncorrelated, for given observables. We hypothesize that the proposed framework for learning informative features results in more discriminative data representation that maintains its generative capabilities. In order to verify our hypothesis we apply the regularization term to the Restricted Boltzmann Machine (RBM) and carry out empirical study on three classification problems: character recognition, object recognition, and document classification. The experiments confirm that the proposed approach indeed increases discriminative and generative performance in comparison to RBM trained without any regularization and with the weight-decay, the sparse regularization, the max-norm regularization, Dropout and Dropconnect.

show abstract

“…The Kullback-Leibler's measure can be understood like a comparison criterion between two distributions. In this section, we derive two classes of entropy measures and one class of divergence measures which can be understood as new goodness-of-fit quantities such those discussed by Seghouane and Amari (2007). All these measures are defined for one element or between two elements in the MOEW family.…”

Section: Information Theory Measuresmentioning

confidence: 99%

The Marshall-Olkin extended Weibull family of distributions

Santos‐Neto

Bourguignon

Zea

et al. 2014

J Stat Distrib Appl

View full text Add to dashboard Cite

We introduce a new class of models called the Marshall-Olkin extended Weibull family of distributions based on the work by Marshall and Olkin (Biometrika 84:641-652, 1997). The proposed family includes as special cases several models studied in the literature such as the Marshall-Olkin Weibull, Marshall-Olkin Lomax, Marshal-Olkin Fréchet and Marshall-Olkin Burr XII distributions, among others. It defines at least twenty-one special models and thirteen of them are new ones. We study some of its structural properties including moments, generating function, mean deviations and entropy. We obtain the density function of the order statistics and their moments. Special distributions are investigated in some details. We derive two classes of entropy and one class of divergence measures which can be interpreted as new goodness-of-fit quantities. The method of maximum likelihood for estimating the model parameters is discussed for uncensored and multi-censored data. We perform a simulation study using Markov Chain Monte Carlo method in order to establish the accuracy of these estimators. The usefulness of the new family is illustrated by means of two real data sets.Mathematics Subject Classification (2010): 60E05; 62F03; 62F10; 62P10

show abstract

The AIC Criterion and Symmetrizing the Kullback–Leibler Divergence

Cited by 134 publications

References 25 publications

Changes in rainfall regime over Burkina Faso under the climate change conditions simulated by 5 regional climate models

Changes in rainfall regime over Burkina Faso under the climate change conditions simulated by 5 regional climate models

Learning Informative Features from Restricted Boltzmann Machines

The Marshall-Olkin extended Weibull family of distributions

Contact Info

Product

Resources

About