Mental Model Assessments: Is There Convergence Among Different Methods?

Abstract

Knowledge elicitation and mental model assessment methods are becoming increasingly popular in applied psychology. However, there continue to be questions about the psychometrics of knowledge elicitation methods. Specifically, more needs to be known regarding the stability and consistency of the results over time (i.e., whether the methods are reliable) and regarding the degree to which the results correctly represent the underlying knowledge structures (i.e., whether the methods are valid).This paper focuses on the convergence among three different assessment methods: (a) pairwise relatedness ratings using Pathfinder, (b) concept mapping, and (c) card sorting. Thirty-six participants completed all three assessments using the same set of twenty driving-related terms. Assessment sequences were counterbalanced, and participants were randomly assigned to one of the six assessment sequences. It was found that the three assessment methods showed very low convergence as measured by the average correlation across the three methods within the same person. Indeed, convergence was lower than the sharedness across participants (as measured by the average correlation across participants within the same assessment method). Additionally, there were order effects among the different assessment sequences. Implications for research and practice are discussed.

Is a Picture Worth a Thousand Words? Investigating Structural Knowledge with Textual and Pictorial Stimuli

“…In this To assess the degree to which different participants had similar mental models of the driving terms (i.e., had shared knowledge structures), the correlation coefficients were calculated for all possible participant pairs and then averaged within each of the three assessment methods and within the two sorting procedures. The resulting average correlations (Table 1) were similar in size with those previously found by Evans, Hitt, and Jentsch (2001). These correlations were then broken down within each condition by sorting structure type and can be seen in Table 2. A two-way ANOVA was conducted.…”

Managing construction risks of AP1000 nuclear power plants in China

“…Researchers have shown that there are inconsistencies between extracted knowledge. For example, a study by Evans et al (2001) shows how different knowledge elicitation techniques produce different results. Hence, it is important to choose the most suitable knowledge elicitation technique with respect to the study objectives and required cognitive aspects.…”

Advancing the practice of cognitive task analysis: a call for taxonomic research

“…Strong empirical work comparing CTA methods has taken place (e.g. Cooke et al 1986, Burton et al 1987, Hoffman 1987, Russo et al 1989, Thorsden 1991, Cooke 1992, Chao and Salvendy 1994, Evans et al 2001, Hoffman et al 2002, but none has attempted to identify causal mechanisms underlying effectiveness or make broader claims about categories of techniques. As a result, they cannot provide any robust generalisations about types of CTA in terms of application or effectiveness, a necessary starting point for the further development of a science of CTA.…”