Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking

Walsh, Cole; Quinn, Katherine N.; Wieman, Carl; Holmes, N. G.

doi:10.1103/physrevphyseducres.15.010135

Cited by 70 publications

(61 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Questions are scored on an approximately continuous scale and bounded between 0 and 1. For a more detailed description of the PLIC including the development and validation process, scoring scheme, and example questions with response choices see [6].…”

Section: A Data Sourcesmentioning

confidence: 99%

“…As PER trends toward new and innovative assessments [5][6][7], researchers are seeking new validation methods. Brewe et al, for example, developed a novel method for examining response choices by employing network analysis to find communities of non-normative responses [8].…”

Section: Introductionmentioning

confidence: 99%

“…We use a concept from information theory, mutual information [9], to examine how much information response choices provide about the underlying factors of an assessment. In the rest of the paper, we introduce mutual information and describe how it can be used to evaluate response choices using the Physics Lab Inventory of Critical thinking (PLIC) [6] as an example. The PLIC uses a multiple response format, but the method described here can be used with any assessment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Assessing the assessment: Mutual information between response choices and factor scores

Walsh¹,

Holmes²

2020

2019 Physics Education Research Conference Proceedings

Self Cite

View full text Add to dashboard Cite

Validated formative assessment tools provide a reliable way to compare student learning across variables such as pedagogy and curricula, or demographics. Such assessments typically employ a closed-response format developed from student responses to open-response questions and interviews with students and experts. The validity and reliability of these assessments is usually evaluated using statistical tools such as classical test theory or item response theory. The suitability of individual questions on an assessment can be examined using either of these methods, but so far little attention has been given to evaluating the suitability of individual response choices available in each question. Here, we use mutual information, a tool rarely used in PER, to quantitatively evaluate the utility of response choices in an assessment. We use the Physics Lab Inventory of Critical thinking (PLIC) as an example to illustrate how assessment developers can use this method for their own assessments. The PLIC was designed to measure three latent constructs and we confirm this structure through a factor analysis. We calculate factor scores that represent performance on each of the proposed constructs and evaluate the suitability of individual response choices in terms of how much information they provide about these factor scores.

show abstract

Section: A Data Sourcesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the assessment: Mutual information between response choices and factor scores

Walsh¹,

Holmes²

2020

2019 Physics Education Research Conference Proceedings

Self Cite

View full text Add to dashboard Cite

show abstract

“…They allow students to engage with concepts and practices that are fundamental to physics, and STEM more generally. As such, education researchers have developed, and are currently developing, many assessment tools that are suited for lab courses, both for instructors to measure and improve their courses, and for researchers to understand the unique learning that occurs in lab settings [1][2][3][4][5].…”

Section: Introductionmentioning

confidence: 99%

Methodological development of a new coding scheme for an established assessment on measurement uncertainty in laboratory courses

Pollard¹,

Hobbs²,

Dounas-Frazer³

et al. 2020

2019 Physics Education Research Conference Proceedings

View full text Add to dashboard Cite

Student understanding around measurement uncertainty is an important learning outcome in physics lab courses across the US, including at the University of Coloroado Boulder (CU), where it is among the major learning outcomes for the large introductory stand-alone physics lab course. One research tool for studying student understanding around measurement uncertainty, which we use in this course, is the Physics Measurement Questionnaire (PMQ), an open-response assessment for measuring student understanding of measurement uncertainty. Interpreting and analyzing PMQ data involves coding students' written explanations to open-response questions. However, the preexisting scoring scheme for the PMQ does not fully capture the breadth and depth of reasoning contained in our students' responses. Therefore, we created a new coding scheme for the PMQ based on responses from our students. Here, we document our process to develop a new coding scheme for the PMQ, and describe the resulting codes. We also present examples of what can be learned from applying the new coding scheme at our institution.

show abstract

“…Recently, physics education researchers have attempted to address both of these challenges by shifting RBAs to online platforms (e.g., [5,[7][8][9]). Hosting the RBAs online allows instructors to assign the RBA for student to complete outside of class, freeing them from the need to sacrifice class time.…”

Section: Introductionmentioning

confidence: 99%

Investigating students’ behavior and performance in online conceptual assessment

Wilcox

Pollock

2019

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

Historically, the implementation of research-based assessments (RBAs) has been a driver of educational change within physics and helped motivate adoption of interactive engagement pedagogies. Until recently, RBAs were given to students exclusively on paper and in-class; however, this approach has important drawbacks including decentralized data collection and the need to sacrifice class time.Recently, some RBAs have been moved to online platforms to address these limitations. Yet, online RBAs present new concerns such as student participation rates, test security, and students' use of outside resources. Here, we report on a study addressing these concerns in both upper-division and lower-division undergraduate physics courses. We gave RBAs to courses at five institutions; the RBAs were hosted online and featured embedded JavaScript code which collected information on students' behaviors (e.g., copying text, printing). With these data, we examine the prevalence of these behaviors, and their correlation with students' scores, to determine if online and paper-based RBAs are comparable. We find that browser loss of focus is the most common online behavior while copying and printing events were rarer. We found that correlations between these behaviors and student performance varied significantly between introductory and upper-division student populations, particularly with respect to the impact of students copying text in order to utilize internet resources. However, while the majority of students engaged in one or more of the targeted online behaviors, we found that, for our sample, none of these behaviors resulted in a significant change in the population's average performance that would threaten our ability to interpret this performance or compare it to paper-based implementations of the RBA.

show abstract

Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking

Cited by 70 publications

References 46 publications

Assessing the assessment: Mutual information between response choices and factor scores

Assessing the assessment: Mutual information between response choices and factor scores

Methodological development of a new coding scheme for an established assessment on measurement uncertainty in laboratory courses

Investigating students’ behavior and performance in online conceptual assessment

Contact Info

Product

Resources

About