Global model analysis by parameter space partitioning.

Abstract

To model behavior, we need to know how models behave. This means learning what other behaviors a model can produce besides the one generated by participants in an experiment. This is a difficult problem because of the complexity of psychological models (e.g., their many parameters) and because the behavioral precision of models (e.g.,interval-scale performance) often mismatches their testable precision in experiments, where qualitative, ordinal predictions are the norm. Parameter space partitioning is a solution that evaluates model performance at a qualitative level. Given a definition of a qualitative data pattern, there exists a partition on the model's parameter space that divides it into regions that correspond to each data pattern.Markov-chain Monte Carlo methods are used to discover and define these regions. Three application examples, all using connectionist models, demonstrate its potential and versatility for studying the global behavior of psychological models. Among other things, one can easily assess how central and robust the empirical data pattern is to the model, as well as the range and characteristics of its other behaviors. Parameter Space Partitioning 3The experimental method of scientific inquiry has proven to work quite well in psychology. Its unique blend of methodological control and statistical inference are effective for testing qualitative (i.e., ordinal) predictions derived from theories of behavior. Data collection and dissemination have become very efficient, so much so that far more may be known about a behavioral phenomenon than is reflected in its corresponding theory.That our knowledge extends beyond the reach of the theory may be a sign of productive science, and underscores the fact that theories are broad conceptualizations about behavior that cannot be expected to explain the minutia in data. Cognitive modeling is a research tool that can act as a counter-force to slow and fill this explanatory gap. It compensates for a theory's limitations of precision in data synthesis, description, and prediction. Whether the model is an implementation of an existing theory, or a neurally inspired environment in which to study processing (e.g., information integration, representational specificity, probabilistic learning), models are rich sources of ideas and information on how to think about perception, cognition, and action. The pros and cons of various implementations can be evaluated. Inconsistencies and hidden assumptions can come to light during model creation and evaluation. In short, the modeler is forced to confront the complexity of what is being modeled, and in the process, can gain insight into the relationship between variables and the functionality of the model (see Shiffrin & Nobel, 1998 for a personal account of this process).Of course, the virtues of modeling are accompanied by vices. One of the more serious, often leveled against connectionist models (Dawson & Shamanski, 1994;McCloskey, 1991) but by no means restricted to them, is that model behavior can be myste...

Are there interactive processes in speech perception?

“…However, subsequent experiments using eye-movement analysis (a finer grained on-line method) and subsequent TRACE simulations using standard parameters found that the TRACE model was consistent with the behavioral data [50]. Analysis of global model behavior [51] confirmed that TRACE produces the correct behavioral pattern; in fact this analysis showed that TRACE provides a more robust fit to the behavioral data in this case than the autonomous Merge model [4].…”

Rational adaptation under task and processing constraints: Implications for testing theories of cognition and action.

“…Model fitting has recently come under much scrutiny (Kieras & Meyer, 2000;Pitt, Kim, Navarro, & Myung, 2006;Roberts & Pashler, 2000), and it is worth considering whether recent methodological prescriptions for comparing data to parametric theories might be used to address the architecture-strategy credit assignment problem. Framed this way, the question is to what extent some given match to data provides support for the structural invariants of a theory with degrees of freedom.…”

Measuring Model Flexibility With Parameter Space Partitioning: An Introduction and Application Example

“…For a thorough presentation of the technical and implementation details of the method, the reader should consult Pitt et al (2006).…”

“…To fill this void, Pitt, Kim, Navarro, and Myung (2006) introduced a model analysis method dubbed parameter space partitioning (PSP). PSP finds all the data patterns that a model can produce in an experimental setting.…”

“…The lower graphs depict the fits of three models to the same data set, increasing in flexibility from left to right (adapted from . Bottom panel: An illustration of the parameter space partitioning analysis for two hypothetical models in an experiment with three conditions: A, B, and C (adapted from Pitt, Kim, Navarro, & Myung, 2006).…”