Assessment of representational competence in kinematics

Abstract

A two-tier instrument for representational competence in the field of kinematics (KiRC) is presented, designed for a standard (1st year) calculus-based introductory mechanics course. It comprises 11 multiple choice (MC) and 7 multiple true-false (MTF) questions involving multiple representational formats, such as graphs, pictures, and formal (mathematical) expressions (1st tier). Furthermore, students express their answer confidence for selected items, providing additional information (2nd tier). Measurement characteristics of KiRC were assessed in a validation sample (pre-and post-test, N ¼ 83 and N ¼ 46, respectively), including usefulness for measuring learning gain. Validity is checked by interviews and by benchmarking KiRC against related measures. Values for item difficulty, discrimination, and consistency are in the desired ranges; in particular, a good reliability was obtained (KR20 ¼ 0.86). Confidence intervals were computed and a replication study yielded values within the latter. For practical and research purposes, KiRC as a diagnostic tool goes beyond related extant instruments both for the representational formats (e.g., mathematical expressions) and for the scope of content covered (e.g., choice of coordinate systems). Together with the satisfactory psychometric properties it appears a versatile and reliable tool for assessing students' representational competency in kinematics (and of its potential change). Confidence judgments add further information to the diagnostic potential of the test, in particular for representational misconceptions. Moreover, we present an analytic result for the question-arising from guessing correction or educational considerations-of how the total effect size (Cohen's d) varies upon combination of two test components with known individual effect sizes, and then discuss the results in the case of KiRC (MC and MTF combination). The introduced method of test combination analysis can be applied to any test comprising two components for the purpose of finding effect size ranges.

Student understanding of graph slope and area under a curve: A replication study comparing first-year physics and economics students

“…However, little is known about the influence of the subject context (e.g., finance or physics) on students' confidence ratings and about the ability to judge one's own performance in the context of graphs. Since confidence ratings are found to be an efficient tool to indicate the existence of misconceptions and learning difficulties [14], the assessment of students' response confidence rating in this study enriches the research about students' misconceptions [8].…”

“…Moreover, it was found that poorly performing students grossly overestimated their performance, which is widely known as the Dunning-Kruger effect [13]. This effect was recently validated in PER [9] and could also be observed in problems testing the representational competence of physics students in the context of kinematics [14]. However, little is known about the influence of the subject context (e.g., finance or physics) on students' confidence ratings and about the ability to judge one's own performance in the context of graphs.…”

“…In this particular case of items, incorrect alternatives were constructed to address common misconceptions and rather simple calculation errors (e.g., determining the area of a rectangle instead of a triangle in the case of the area questions). Therefore, the participants were able to select an incorrect alternative that reflected their flawed thinking process, resulting in a high response confidence [14]. It can be assumed that the mathematical procedures for solving the questions are more present for the physics than for the economics students [7], enabling physics students to reflect on their answers more deeply whereas the metacognitive insight is reduced for economics students.…”

Testing students ability to use derivatives, integrals, and vectors in a purely mathematical context and in a physical context

“…Representational fluency (i.e., the ability to dynamically pass from one representational format of a concept to another) is very important in physics and it is considered a sign of expertise [72,73]. Specific instruments for assessing students' representational fluency have been designed, such as the KiRC inventory for kinematics [74], and, on a more interdisciplinary perspective, the representational fluency survey [75].…”

Visual attention while solving the test of understanding graphs in kinematics: an eye-tracking analysis

“…For example, the TUG-K has been used as a preand posttest to investigate the effectiveness of instructions (e.g., video-based motion analysis [5]) or to study the relationship between the understanding of kinematics graphs and other variables (e.g., gender) [6]. The test has also been used as a reference to design new, related tests: for example, a test to assess the understanding of graphs in the context of calculus [2], the kinematics concept test [7] or the kinematics representational competence inventory [8]. For the TUG-K, the graphs of all test items relate to motion in one dimension and the items address the concepts of graph slope and area under the curve for different objectives [1].…”