Development and Validation of a Science Inquiry Skills Assessment

The Psychometric Modeling of Scientific Reasoning: a Review and Recommendations for Future Avenues

“…This number generally tended to be equal or lower than that of theoretically conceptualized components. In three cases, the number of modeled dimensions did not at all overlap with the number of theoretically conceptualized components of scientific reasoning (Esswein, 2010;Lou et al, 2015;Mayer et al, 2014). Thus, in these cases, the researchers estimated psychometric models that did not correspond to their theoretical deliberations.…”

“…These components were described as sub-competencies (Grube, 2010), primary skills (Lou, Blanchard, & Kennedy, 2015), ability components (Körber et al, 2014;Kuo, Wu, Jen, & Hsu, 2015), and processes consisting of several skills (Nowak, Nehring, Tiemann, & Upmeier zu Belzen, 2013). In four studies, scientific reasoning was conceptualized to consist of one such general component, in six studies four components were conceptualized, in one study six components (Lou et al, 2015), and in three of the studies both one general component and alternatively three (Nowak et al, 2013), five (Körber et al, 2014), or seven components (Hartmann et al, 2015) were conceptualized (Table 1). There is thus no consensus how many components scientific reasoning has-neither between, nor within studies.…”

“…In each of the studies, researchers employed a variety of principled validity measures in order to ensure the quality of the respective items and instrument. Studies conducted in the United States (e.g., Kuo et al, 2015;Lou et al, 2015) reasoning within a specific domain and rooted their psychometric approaches within the assessment of competencies (e.g., Kremer et al, 2014;Hartmann et al, 2015). Overall, in each study discussed frameworks for the assessment of scientific reasoning and various measures to ensure validity.…”

“…Some studies also conducted differential item functioning (DIF) analyses to examine whether item difficulty parameters are comparable between subgroups-an important characteristic of invariant measurement (Hambleton, Swaminathan, & Rogers, 1991;Siersma & Eusebi, 2013). Furthermore, in one study (Lou et al, 2015), residual correlations between items (local dependence analysis) and a principal component analysis were applied to examine the strength of systematic residual variance in the data. To summarize, the studies' overall repertoire of fit investigation encompassed infit and outfit statistics for items and persons, DIF analysis, residual and dimensionality analysis; most studies, however, mainly or exclusively relied on item infit statistics.…”

“…In the reviewed literature, researchers made inferences regarding both the statistical and the psychological structure of scientific reasoning. For example, Nowak et al 2013 Interpreting this single overarching trait, Lou et al (2015) suggested that researchers and teachers use a single score from the instrument and interpret subscale scores with caution. Kuo et al (2015) found that intercorrelations between four psychometric dimensions were higher than the individual dimensions' reliabilities.…”