On the Use of Entropy to Improve Model Selection Criteria

Murari, A.; Peluso, E.; Cianfrani, Francesco; Gaudio, P.; Lungaroni, M.

doi:10.3390/e21040394

Cited by 26 publications

(23 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Equations (7) and (8) suggested also a consideration about the asymptote stability of AIC GF and BIC GF (where GF means Goodness of Fit); if the model was perfect, the Z score would tend to zero with increasing the number of points, and the AIC GF and BIC GF would converge to AIC H and BIC H . The asymptotic converge of the criteria using the entropy of the residuals has already been demonstrated in [9], proving the asymptotic convergence of AIC GF and BIC GF as well. The AIC GF and BIC GF of Equations (7) and (8) were, therefore, the new versions of the model selection criteria that had been tested and whose performance is described in the rest of the paper.…”

Section: Model Selection and Goodness Of Fit Testsmentioning

confidence: 59%

“…For Gaussian and flat distributed noise, all versions of both AIC and the BIC (traditional formulation, Shannon entropy, and goodness of fit) returned basically equal values in all cases. As shown in [9], for these cases, the upgraded versions-AIC H , BIC H and AIC GF , BIC GF -improve only the convergence to the right model in limited and very difficult cases. For the other two types of noise, the situation was dramatically different.…”

Section: Results For Exponential Functions Polynomials and Power Lawsmentioning

confidence: 94%

“…while the function for the fit was the one given in Equation (9). A plot of the original function and with added noise of Poisson distribution is shown in Figure 2.…”

Section: Results For Exponential Functions Polynomials and Power Lawsmentioning

confidence: 99%

“…These new versions of the criteria clearly outperformed the traditional version of the AIC and BIC, and they had also asymptotic convergence [9]. On the other hand, entropy is a quite crude indicator of the residual distributions.…”

Section: Model Selection and Goodness Of Fit Testsmentioning

confidence: 96%

“…The practical importance of this question is not to be underestimated because, in many applications, the assumptions leading to expressions (3) and (4) are clearly violated. The first attempt in this direction, reported in [9], relied on the Shannon entropy as an indicator of the residual distribution. The encouraging results, obtained with this upgrade, has motivated the insertion of more sophisticated summary statistics of the distribution of the residuals into the AIC and BIC, to see whether they could further increase their performance.…”

Section: Introduction To Model Selection Criteria Based On Bayes and mentioning

confidence: 99%

See 4 more Smart Citations

Upgrading Model Selection Criteria with Goodness of Fit Tests for Practical Applications

Rossi

Murari

Gaudio

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

The Bayesian information criterion (BIC), the Akaike information criterion (AIC), and some other indicators derived from them are widely used for model selection. In their original form, they contain the likelihood of the data given the models. Unfortunately, in many applications, it is practically impossible to calculate the likelihood, and, therefore, the criteria have been reformulated in terms of descriptive statistics of the residual distribution: the variance and the mean-squared error of the residuals. These alternative versions are strictly valid only in the presence of additive noise of Gaussian distribution, not a completely satisfactory assumption in many applications in science and engineering. Moreover, the variance and the mean-squared error are quite crude statistics of the residual distributions. More sophisticated statistical indicators, capable of better quantifying how close the residual distribution is to the noise, can be profitably used. In particular, specific goodness of fit tests have been included in the expressions of the traditional criteria and have proved to be very effective in improving their discriminating capability. These improved performances have been demonstrated with a systematic series of simulations using synthetic data for various classes of functions and different noise statistics.

show abstract

Section: Model Selection and Goodness Of Fit Testsmentioning

confidence: 59%

Section: Results For Exponential Functions Polynomials and Power Lawsmentioning

confidence: 94%

“…while the function for the fit was the one given in Equation (9). A plot of the original function and with added noise of Poisson distribution is shown in Figure 2.…”

Section: Results For Exponential Functions Polynomials and Power Lawsmentioning

confidence: 99%

Section: Model Selection and Goodness Of Fit Testsmentioning

confidence: 96%

Section: Introduction To Model Selection Criteria Based On Bayes and mentioning

confidence: 99%

See 3 more Smart Citations

Upgrading Model Selection Criteria with Goodness of Fit Tests for Practical Applications

Rossi

Murari

Gaudio

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

Oreamuno

Oladyshkin

Nowak

2022

Preprint

View full text Add to dashboard Cite

Bayesian model selection (BMS) and Bayesian model justifiability analysis (BMJ) provide a statistically rigorous framework to compare competing conceptual models through the use of Bayesian model evidence (BME). However, BME-based analysis has two main limitations: (1) it's powerless when comparing models with different data set sizes and/or types of data and(2) doesn't allow to judge a model's performance based on its posterior predictive capabilities. Thus, traditional BME-based approaches ignore useful data or models due to issue (1) or disregards Bayesian updating because of issue (2). To address these limitations, we advocate to include additional information-theoretic scores into BMS and BMJ analysis: expected log-predictive density (ELPD), relative entropy (RE) and information entropy (IE). Exploring the connection between Bayesian inference and information theory, we explicitly link BME and ELPD together with RE and IE to indicate the information flow in BMS and BMJ analysis. We show how to compute and interpret these scores alongside BME, and apply it in a model selection and similarity analysis framework. We test the methodology on a controlled 2D groundwater setup considering five competing conceptual models accompanied with different data sets. The results show how the information-theoretic scores complement BME by providing a more complete picture concerning the Bayesian updating process. Additionally, we present how both RE and IE can be used to objectively compare models that feature different data sets. Overall, the introduced Bayesian information-theoretic framework helps to avoid any potential loss of information and leads to an informed decision for model selection and similarity.

show abstract

Detection of interannual ensemble forecast signals over the North Atlantic and Europe using atmospheric circulation regimes

Falkena

Wiljes

Weisheimer

et al. 2021

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

To study the forced variability of atmospheric circulation regimes, the use of model ensembles is often necessary for identifying statistically significant signals as the observed data constitute a small sample and are thus strongly affected by the noise associated with sampling uncertainty. However, the regime representation is itself affected by noise within the atmosphere, which can make it difficult to detect robust signals. To this end we employ a regularised k-means clustering algorithm to better identify the signal in a model ensemble. The approach allows for the identification of six regimes for the wintertime Euro-Atlantic sector and leads to more pronounced regime dynamics, compared to results without regularisation, both overall and on sub-seasonal and interannual time-scales. We find that sub-seasonal variability in the regime occurrence rates is mainly explained by changes in the seasonal cycle of the mean climatology. On interannual time-scales relations between the occurrence rates of the regimes and the El Niño Southern Oscillation (ENSO) are identified. The use of six regimes captures a more detailed response of the circulation to ENSO compared to the common use of four regimes. Predictable signals in occurrence rate on interannual time-scales are found for the two zonal flow regimes, namely a regime consisting of a negative geopotential height anomaly over the Norwegian Sea and Scandinavia, and the positive phase of the NAO. The signal strength for these regimes is comparable between observations and model, in contrast to that of the NAO-index where the signal strength in the observations is underestimated by a factor of 2 in the model. Our regime analysis suggests that this signal-to-noise problem for the NAO-index is primarily related to those atmospheric flow patterns associated with the negative NAO-index as we find poor predictability for the corresponding NAO− regime.

show abstract

On the Use of Entropy to Improve Model Selection Criteria

Cited by 26 publications

References 19 publications

Upgrading Model Selection Criteria with Goodness of Fit Tests for Practical Applications

Upgrading Model Selection Criteria with Goodness of Fit Tests for Practical Applications

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

Detection of interannual ensemble forecast signals over the North Atlantic and Europe using atmospheric circulation regimes

Contact Info

Product

Resources

About