Bayesian and frequentist analysis of True and Error models

Abstract

Birnbaum and Quispe-Torreblanca (2018) presented a frequentist analysis of a family of six True and Error (TE) models for the analysis of two choice problems presented twice to each participant. Lee (2018) performed a Bayesian analysis of the same models, and found very similar parameter estimates and conclusions for the same data. He also discussed some potential differences between Bayesian and frequentist analyses and interpretations for model comparisons. This paper responds to certain points of possible controversy regarding model selection that attempt to take into account the concept of flexibility or complexity of a model. Reasons to question the use of Bayes factors to decide among models differing in fit and complexity are presented. The partially nested inter-relations among the six TE models are represented in a Venn diagram. Another view of model complexity is presented in terms of possible sets of data that could fit a model rather than in terms of possible sets of parameters that do or do not fit a given set of data. It is argued that less complex theories are not necessarily more likely to be true, and when the space of all possible theories is not well-defined, one should be cautious in interpreting calculated posterior probabilities that appear to prove a theory to be true.

Reanalysis of Butler and Pogrebna (2018) using true and error model

“…One can also use Bayesian methods to analyze TE models to calculate posterior distributions of the parameters, given suitable priors. Although arguments can be made for possible advantages or disadvantages of these various techniques, analyses of empirical examples so far have yielded very similar solutions and conclusions from the different methods (Lee, 2018;Birnbaum, 2019;Birnbaum & Quispe-Torreblanca, 2018;Schramm, 2020). In this paper, both TE8x8_fit.xlsx and TE8x2_fit.R were applied, minimizing…”

“…I prefer to compare theories by their success in predicting the results of new tests of diagnostic properties, where the rival theories make qualitatively different predictions. Further comments on testing and comparison of models like these are in Birnbaum (2019).…”

MARTER: Markov True and Error model of drifting parameters

“…So, rather than assume that iid holds, that errors are of a certain arbitrary magnitude, or that all choice problems have the same error rate, one can test these assumptions in the MARTER framework. An example comparing six models (with three nested TE models that impose various assumptions concerning errors and in each of which EU is a special case)is presented in Birnbaum and Quispe-Torreblanca (2018) and discussed further from a Bayesian perspective in Lee (2019) and Birnbaum (2019).…”

“…Bhatia and Loomes (2017) explore the effects of random variation in parameters of a decision making model, but note that such variation cannot explain all of the phenomena of choice data, so there must be another source of error. This article will expand upon an approach based on True and Error (TE) theory (Birnbaum, 2011(Birnbaum, , 2013(Birnbaum, , 2019Birnbaum & Bahra, 2007a, 2012a, 2012bBirnbaum & Diecidue, 2015;Birnbaum & Gutierrez, 2007;Birnbaum & Schmidt, 2008;Birnbaum, Schmidt & Schneider, 2017;Birnbaum & Quispe-Torreblanca, 2018). This approach does allow separation (and estimations) of the variability due to changes in true preferences and due to random errors.…”

True and error analysis instead of test of correlated proportions: Can we save lexicographic semiorder models with error theory?

“…The null hypothesis, H0thinmathspace:thinmathspacepSR′=pRS′=0 is a special case of each of these three TE models. Birnbaum (2019) presented a figure to show the nesting relationships among these six possible models.…”