Assessing the distinguishability of models and the informativeness of data

Navarro, Daniel J.; Pitt, Mark A.; Myung, In Jae

doi:10.1016/j.cogpsych.2003.11.001

Cited by 80 publications

(95 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The qualitative and quantitative predictions of the mixtureof-processes hypothesis were compared to two existing And one final remark: Note that the two-stage-processing model had the fewest parameters (12, rather than 13 for the bound-change hypothesis and 14 for the mixture-ofprocesses hypothesis) but gave the best account of the data. As is well known (see, e.g., Navarro, Pitt, & Myung, 2004;Wagenmakers, Ratcliff, Gomez, & Iverson, 2004), the number of parameters is not the only criterion for model comparison. In principle, one hypothesis or model might well be more flexible than others, and therefore fitting the model to data would be easier for that model than for the others.…”

Section: Do Different Experimental Setups Influence Choice Proportions?mentioning

confidence: 99%

A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks

Diederich

2008

Perception & Psychophysics

View full text Add to dashboard Cite

Section: Do Different Experimental Setups Influence Choice Proportions?mentioning

confidence: 99%

A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks

Diederich

2008

Perception & Psychophysics

View full text Add to dashboard Cite

“…What is the purpose, the reviewer wondered, of comparing the models if we already know that they are wrong? The question speaks to some important issues in model development and testing (for discussions of other issues in model testing, see Navarro, Pitt & Myung 2004;Pitt & Myung 2002).…”

Section: Model Predictions With Backward Maskingmentioning

confidence: 99%

Testing models of object substitution with backward masking

Francis

Cho

2007

Perception & Psychophysics

View full text Add to dashboard Cite

“…Model comparison using model mimicry simulations Wagenmakers et al (2004) presented a general method to quantify model mimicry, termed the parametric bootstrap cross-fitting method, PBCM (see Navarro, Pitt, & Myung, 2004, for a similar technique). Consider a comparison between Model A and Model B.…”

mentioning

confidence: 99%

The diagnosticity of individual data for model selection: Comparing signal-detection models of recognition memory

2011

View full text Add to dashboard Cite

We tested whether the unequal-variance signaldetection (UVSD) and dual-process signal-detection (DPSD) models of recognition memory mimic the behavior of each other when applied to individual data. Replicating previous results, there was no mimicry for an analysis that fit each individual, summed the goodness-of-fit values over individuals, and compared the two sums (i.e., a single model selection). However, when the models were compared separately for each individual (i.e., multiple model selections), mimicry was substantial. To quantify the diagnosticity of the individual data, we used mimicry to calculate the probability of making a model selection error for each individual. For nondiagnostic data (high model selection error), the results were compatible with equalvariance signal-detection theory. Although neither model was justified in this situation, a forced-choice between the UVSD and DPSD models favored the DPSD model for being less flexible. For diagnostic data (low model selection error), the UVSD model was selected more often.Keywords Model mimicry . Model flexibility . Recognition memory . Unequal-variance signal-detection model . Dual-process signal-detection model When comparing models based on goodness of fit (GOF), model flexibility (or complexity) is an important issue to address. Model flexibility refers to the ability of a model to flexibly capture any data pattern (Myung, 2000). A useful concept for understanding model flexibility is the response surface methodology (Bates & Watts, 1988), which is a plot of all possible results that a model can explain (an area in the data space). A more flexible model will cover a larger proportion of the data space, which indicates that it is difficult to find data to reject that model, as compared to all possible alternative models. However, sometimes researchers want to directly compare two leading candidate models. The relevant consideration is the area of overlap between the two models (the region in which they mimic each other), as compared to the area in the data space that is unique to each model, and in this case, model mimicry is an important consideration (Wagenmakers, Ratcliff, Gomez, & Iverson, 2004). In contrast to absolute flexibility, model mimicry can be thought of as a measure of relative flexibility when comparing two particular models.Another important issue to consider when comparing two models is whether to analyze data at the individual or the group level. Cohen, Sanborn, and Shiffrin (2008) conducted model mimicry simulations fitting both individual and group data in order to determine which method was more effective in recovering the true underlying model (see also Cohen, Rotello, & Macmillan, 2008). They found that model selection based on the sum of GOF values was more accurate when the data of each individual were fit separately, provided that there were a sufficient number of data for each individual; otherwise, model selection was superior when based on group data. The experiments we analyzed collected at least 140 obse...

show abstract

Assessing the distinguishability of models and the informativeness of data

Cited by 80 publications

References 55 publications

A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks

A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks

Testing models of object substitution with backward masking

The diagnosticity of individual data for model selection: Comparing signal-detection models of recognition memory

Contact Info

Product

Resources

About