Using a Concept Inventory to Assess the Reasoning Component of Citizen-Level Science Literacy: Results from a 17,000-Student Study

Abstract

After articulating 12 concepts for the reasoning component of citizen-level science literacy and restating these as assessable student learning outcomes (SLOs), we developed a valid and reliable assessment instrument for addressing the outcomes with a brief 25-item science literacy concept inventory (SLCI). In this paper, we report the results that we obtained from assessing the citizen-level science literacy of 17,382 undergraduate students, 149 graduate students, and 181 professors. We address only findings at or above the 99.9% confidence level. We found that general education (GE) science courses do not significantly advance understanding of science as a way of knowing. However, the understanding of science’s way of knowing does increase through academic ranks, indicating that the extended overall academic experience better accounts for increasing such thinking capacity than do science courses alone. Higher mean institutional SLCI scores correlate closely with increased institutional selectivity, as measured by the institutions’ higher mean SAT and ACT scores. Socioeconomic factors of a) first-generation student, b) English as a native language, and c) interest in commitment to a science major are unequally distributed across ethnic groups. These factors proved powerful in accounting for the variations in SLCI scores across ethnicities and genders.

Paired Measures of Competence and Confidence Illuminate Impacts of Privilege on College Students

“…1 Our goals for this study are three-fold and aligned with our data collection strategies. First, we aim to confirm our earlier work (Favazzo et al 2014;Nuhfer et al 2016aNuhfer et al , 2016bNuhfer et al , and 2017) that the relationship between people's selfassessment and their actual competence is meaningful and that most people are adequate self-assessors. We use a competency measure as expressed by participants' scores on a special kind of multiple-choice test called a concept inventory.…”

“…The construction of the paired instruments, the conditions under which they collect data, and the psychometrics of both are already detailed in Nuhfer et al (2016a;2016b;. We do not repeat that detail here.…”

“…Inadequate self-assessment is inaccuracy greater than ±20 ppts inaccuracy but less than or equal to ±30 ppts, and extreme self-assessment inaccuracy defines misestimates as greater than ± 30 ppts. Nuhfer et al (2016a) noted the difficulty of using language to describe the precise nature of what is being reported through the numerical expression of selfassessed competence. There is no doubt that an individual's self-assessed rating of competence is a felt affective response, but this affective feeling is one largely informed by the cognitive knowledge and experience related to the specific content relative to the challenge being assessed.…”

“…These selfratings express affective self-perceived competence and are informed to varying degrees by awareness of actual competence and prior experiences from scores obtained through competency testing. In our recent work (Nuhfer 2015;Nuhfer et al 2016a and, we emphasized that a requisite for the validity of any such studies is that participants must clearly understand the challenges on which they are being asked to self-assess their competence. Both the measures of demonstrated competence and self-assessed competence must come from instruments that provide data of known reliability before one should attempt to interpret this paired data numerically.…”

Assessing Student Learning and Improving Instruction with Transition Matrices

“…The transition matrices approach differentiates large FCI gains as a result of instruction versus smaller gains (or losses)-in either case the transition matrices would look quite different but the transition probabilities matrices may be approximately the same. In future work we will perform a similar analysis of determining transition probabilities for a larger sample both on the FCI and also on the Science Literacy Concept Inventory [9].…”