A Practical, Accurate, Information Criterion for Nth Order Markov Processes

Abstract

The recent increase in the breath of computational methodologies has been matched with a corresponding increase in the difficulty of comparing the relative explanatory power of models from different methodological lineages. In order to help address this problem a Markovian information criterion (MIC) is developed that is analogous to the Akaike information criterion (AIC) in its theoretical derivation and yet can be applied to any model able to generate simulated or predicted data, regardless of its methodology. Both the AIC and proposed MIC rely on the Kullback-Leibler (KL) distance between model predictions and real data as a measure of prediction accuracy. Instead of using the maximum likelihood approach like the AIC, the proposed MIC relies instead on the literal interpretation of the KL distance as the inefficiency of compressing real data using modelled probabilities, and therefore uses the output of a universal compression algorithm to obtain an estimate of the KL distance. Several Monte Carlo tests are carried out in order to (a) confirm the performance of the algorithm and (b) evaluate the ability of the MIC to identify the true data-generating process from a set of alternative models.

Empirical validation of simulated models through the GSL-div: an illustrative application

“…In many cases, this amounts at replicating the largest possible number of stylized facts characterizing the phenomenon of interest (see Dosi et al, 2010Dosi et al, , 2013Dosi et al, , 2015 for business cycle properties, credit and interbank markets or Pellizzari and Forno, 2006;Jacob Leal et al, 2015 for financial markets). Recent attempts are trying to enrich empirical validation going beyond simple replication of empirical regularities, thereby requesting models to generate series that exhibit the same dynamics (Marks, 2013;Lamperti, 2015), conditional probabilistic structure (Barde, 2015) and causal relations (Guerini and Moneta, 2016) as those observed in the real world data. At least partially, such contributions have been motivated by the unsatisfactory results delivered by calibration.…”

Bayesian estimation of agent-based models

“…Recently, Kukacka and Barunik (2016) have proposed non-parametric simulated maximum likelihood, and apply it to the estimation of the model described in Brock and Hommes (1998). Other contributions have introduced sophisticated validation methods aimed at measuring the distance between real and simulated data, where the simulated data can either come from the original AB model (Fabretti, 2013;Marks, 2013;Lamperti, 2015;Recchioni et al, 2015;Barde, 2016;Guerini and Moneta, 2016), or from a surrogate model (Salle and Yildizoglu, 2014;Sani et al, 2016). Minimisation of such distance metrics within a SMD approach leads to alternative estimators with respect to those commonly employed in the literature, such as MSM or II, although the properties of these estimators, including consistency, and their associated uncertainty have yet to be fully assessed.…”

An Empirical Validation Protocol for Large-Scale Agent-Based Models

“…The Markov Information Criterion (MIC) is a recent model comparison methodology developed in Barde (2016a) that provides a measurement of the cross entropy between a model and an empirical data set for any model reducible to a Markov process of arbitrary order. In an analogous manner to a traditional information criterion (AIC, BIC, etc.…”

“…The CTW algorithm is proven to provide an optimal learning performance, in the sense that it achieves the theoretical lower bound on the learning error. As explained in Barde (2016a), this means that a bound correction procedure can be applied to the raw CTW score to correct the measurement error due to learning, thus enabling an accurate measurement of the informational distance between the model and the data. By considering the model as an input/output response function, i.e.…”