Seeking missing pieces in science concept assessments: Reevaluating the Brief Electricity and Magnetism Assessment through Rasch analysis

Abstract

Discipline-based science concept assessments are powerful tools to measure learners' disciplinary core ideas. Among many such assessments, the Brief Electricity and Magnetism Assessment (BEMA) has been broadly used to gauge student conceptions of key electricity and magnetism (E&M) topics in college-level introductory physics courses. Differing from typical concept inventories that focus only on one topic of a subject area, BEMA covers a broad range of topics in the electromagnetism domain. In spite of this fact, prior studies exclusively used a single aggregate score to represent individual students' overall understanding of E&M without explicating the construct of this assessment. Additionally, BEMA has been used to compare traditional physics courses with a reformed course entitled Matter and Interactions (M&I). While prior findings were in favor of M&I, no empirical evidence was sought to rule out possible differential functioning of BEMA that may have inadvertently advantaged M&I students. In this study, we used Rasch analysis to seek two missing pieces regarding the construct and differential functioning of BEMA. Results suggest that although BEMA items generally can function together to measure the same construct of application and analysis of E&M concepts, several items may need further revision. Additionally, items that demonstrate differential functioning for the two courses are detected. Issues such as item contextual features and student familiarity with question settings may underlie these findings. This study highlights often overlooked threats in science concept assessments and provides an exemplar for using evidence-based reasoning to make valid inferences and arguments.

Innovations in Physics Education: Impact of Inquiry-based Laboratory Format on Students’ Perceptions and Learning Gains

“…This is a basic fact of EM, but the act of discovering vs. simply being told "it is so," likely increases the impact this has on a students' perception. With this premise in mind, all Physics 261 students were given the BEMA; designed as a tool for measuring introductory students' conceptual knowledge of EM, BEMA is purported to be a useful component of PER (Ding, 2014;Pollock & Finkelstein, 2014;Ding et al, 2006). Results for this study (for various reasons discussed below) were not as conclusive as hoped for, but certainly substantial enough to warrant further research.…”

“…The conceptual complexity of the subject matter can be particularly challenging for physics students, especially during the second semester, when electricity and magnetism (EM) is the main focus. To accurately assess their perceptions and learning gains, two key elements are necessary: (a) collection of student feedback through self-assessment, focus groups, or individual interview; and (b) assessment of knowledge and skills through written or practical exams or exercises (Lindsey & Nagel, 2015;Zwicki, Hirokawa, Finkelstein, & Lewandowski, 2014;Thacker et al, 2014;Ding, 2014;Ding, Chabay, Sherwood, & Beichner, 2006;Seymour, Wiese, Hunter, & Daffinrud, 2000). Additionally, the importance of ensuring an adequate (and current) knowledge base for instructor's, their ability to assess students' levels of comprehension and explain concepts in a meaningful and productive way has been illustrated (Eylon & Bagno, 2006).…”

“…To measure students' learning gains, the BEMA was administered, during lecture class at the beginning of the semester, and during lab class at the end of the semester (Ding, 2014;Ding et al, 2006). Students did not receive extra credit for completion of this assessment as it was administered as part of the curricula, to help lecture professors gauge student progress.…”

Development and uses of upper-division conceptual assessments

“…Two common perspectives on test development are Classical Test Theory (CTT) [32] and Item Response Theory (IRT) [33]. The majority of conceptual assessments in physics, at both the introductory and upper-division levels have been validated using CTT, while only a small number have been developed or analyzed using IRT [34][35][36][37]. One significant drawback of CTT is that all test statistics are population dependent.…”

Class Tournament as an Assessment Method in Physics Courses: A Pilot Study

“…These include, in particular, different forms of a written test, extensively described and compared in the literature, such as free-and multipleresponse tests (Wilcox & Pollock, 2014), concept tests (such as the Test of Understanding Graphs in Kinematics (Maries & Singh, 2013), Force Concept Inventory (Hestenes et al, 1992) or Brief electricity and magnetism assessment (Ding et al, 2006) and others (Hitt et al, 2014;, constructed-response tests (Slepkov & Shiell, 2014), essay tests (Kruglak, 1955), laboratory skills tests (Doran et al, 1993) and others. Also, many modifications and extensions of these tests have already been proposed in the literature, improving upon their original form (Ding, 2014;Docktor et al, 2015;Wooten et al, 2014;Zwolak & Manogue, 2015). On the other hand, some authors propose to blend formative and summative assessment techniques.…”