Sometimes Less Is Not Enough: A Commentary on Greiff et al. (2015)

Abstract

In this commentary, I discuss some critical issues in the study by Greiff, S.; Stadler, M.; Sonnleitner, P.; Wolff, C.; Martin, R., “Sometimes less is more: Comparing the validity of complex problem solving measures”, Intelligence 2015 , 50 , 100–113. I conclude that—counter to the claims made in the original study—the specific study design was not suitable for deriving conclusions about the validity of different complex problem-solving (CPS) measurement approaches. Furthermore, a more elaborate consideration of previous CPS research was found to challenge Greiff et al.’s conclusions even further. Therefore, I argue that researchers should be aware of the differences between several kinds of CPS assessment tools and conceptualizations when the validity of CPS assessment tools is examined in future research.

Impact of Cognitive Abilities and Prior Knowledge on Complex Problem Solving Performance – Empirical Results and a Plea for Ecologically Valid Microworlds

“…For the latter, the quality of one’s hypotheses, which is based on domain knowledge, is a necessary prerequisite for successfully exploring the system. In summary, the features of MCS measurements outlined here, along with further criticisms of this approach (e.g., Funke, 2014; Scherer, 2015; Schoppek and Fischer, 2015; Dörner and Funke, 2017; Funke et al, 2017; Kretzschmar, 2017), substantially narrow the validity of the MCS approach as an indicator of CPS.…”

“…On the other hand, the relevance of the MCS approach is shown by many studies that have modeled the internal structure of MCS tasks (e.g., Greiff et al, 2012; Sonnleitner et al, 2012), provided evidence that performance variance cannot be sufficiently explained by reasoning (e.g., Wüstenberg et al, 2012; Sonnleitner et al, 2013; Kretzschmar et al, 2016), found strong convergent validity as well as a lower correlation with a CRS (i.e., Tailorshop ; Greiff et al, 2015b; for a different view, see Kretzschmar, 2017), and demonstrated incremental validity in predicting school grades beyond reasoning (e.g., Greiff et al, 2013b; Sonnleitner et al, 2013; for different results, see Kretzschmar et al, 2016; Lotz et al, 2016) and beyond a CRS task (Greiff et al, 2015b). MCS have been proposed as a tool for assessing 21st Century skills (Greiff et al, 2014) and were applied in the international large-scale study PISA to assess general problem-solving skills (OECD, 2014).…”

“…Studies provide support for the psychometric quality, especially the reliability, of the MCS approach, although scale building and some statistics have been criticized (Funke et al, 2017; Kretzschmar, 2017). Only one study so far has attempted to compare MCS and CRS.…”

“…The knowledge scales assessed after exploring the system were used as predictors for the MCS. However, system-specific knowledge for Tailorshop after controlling the system was not assessed (Kretzschmar, 2017). Instead, control performance was used as a predictor of school grades.…”

“…This approach has recently become prominent in educational psychology (e.g., Greiff et al, 2013b; Sonnleitner et al, 2013; Kretzschmar et al, 2014; OECD, 2014; Csapó and Molnár, 2017). In addition, this shift has led to the question of what are and are not complex problems, with some researchers questioning the relevance of MCS as a tool for CPS research and the validity of the conclusions drawn from them (e.g., Funke, 2014; Dörner and Funke, 2017; Funke et al, 2017; Kretzschmar, 2017). …”