Investigating students’ behavior and performance in online conceptual assessment

Abstract

Historically, the implementation of research-based assessments (RBAs) has been a driver of educational change within physics and helped motivate adoption of interactive engagement pedagogies. Until recently, RBAs were given to students exclusively on paper and in-class; however, this approach has important drawbacks including decentralized data collection and the need to sacrifice class time.Recently, some RBAs have been moved to online platforms to address these limitations. Yet, online RBAs present new concerns such as student participation rates, test security, and students' use of outside resources. Here, we report on a study addressing these concerns in both upper-division and lower-division undergraduate physics courses. We gave RBAs to courses at five institutions; the RBAs were hosted online and featured embedded JavaScript code which collected information on students' behaviors (e.g., copying text, printing). With these data, we examine the prevalence of these behaviors, and their correlation with students' scores, to determine if online and paper-based RBAs are comparable. We find that browser loss of focus is the most common online behavior while copying and printing events were rarer. We found that correlations between these behaviors and student performance varied significantly between introductory and upper-division student populations, particularly with respect to the impact of students copying text in order to utilize internet resources. However, while the majority of students engaged in one or more of the targeted online behaviors, we found that, for our sample, none of these behaviors resulted in a significant change in the population's average performance that would threaten our ability to interpret this performance or compare it to paper-based implementations of the RBA.

Designing upper-division thermal physics assessment items informed by faculty perspectives of key content coverage

“…The online administration is motivated by allowing more institutions to be used for piloting and streamlined analysis. We have little concern about students potentially using online resources to perform better on the assessment, as answers to the items will not be found easily via online search and a recent study shows certain online behaviors that may be linked to using internet resources do not appear to meaningfully influence average student performance on similar research-based assessments [54]. Additionally, we advise, in line with best practices from the assessment literature [55,56], that instructors only provide participation credit to students for completion of the assessment (as opposed to assigning scores based on correctness).…”

Online administration of a reasoning inventory in development

“…Though our focus was on in-person, proctored administration of the assessment, we began to consider whether online, unproctored administration would better support validation and wide-spread dissemination. While existing research suggests little or no significant difference in student performance between proctored and unproctored administrations of some RBAs [2][3][4], researchers recommend that online, unproctored administration be validated separately [2]. We wanted to determine whether our instrument could be administered online and unproctored by instructors who were reluctant or unable to allocate class time for administration.…”

“…RBAs can be time consuming to administer in class (both in terms of class time and in processing the data collected), resulting in interest in online administration. Research suggests that online administration of RBAs originally designed to be administered in person largely does not affect student performance [2][3][4]. However, some work suggests that online administration may result in reduced test security (with a small percentage of students copying/printing test items) and more frequent "loss of focus"-that is, students may open other browser windows while completing the assessment [3,4].…”

“…Research suggests that online administration of RBAs originally designed to be administered in person largely does not affect student performance [2][3][4]. However, some work suggests that online administration may result in reduced test security (with a small percentage of students copying/printing test items) and more frequent "loss of focus"-that is, students may open other browser windows while completing the assessment [3,4]. For commonly used or well-known RBAs such as the Force Concept Inventory (FCI) or Brief Electricity and Magnetism Assessment (BEMA), students may be able to find correct answers online, though this does not prevent comparisons with results from in-person administration [4].…”

“…However, some work suggests that online administration may result in reduced test security (with a small percentage of students copying/printing test items) and more frequent "loss of focus"-that is, students may open other browser windows while completing the assessment [3,4]. For commonly used or well-known RBAs such as the Force Concept Inventory (FCI) or Brief Electricity and Magnetism Assessment (BEMA), students may be able to find correct answers online, though this does not prevent comparisons with results from in-person administration [4]. Decreases in student participation rates sometimes seen with online administration can be ameliorated by frequent reminders from instructors, and by following best practices similar to those for in-person administration, such as awarding credit based on completion rather than correctness.…”