Comparing Online and Offline Administration of Multiple Choice Question Assessments to Psychology Undergraduates: Do Assessment Modality or Computer Attitudes Influence Performance?

Abstract

Recent interest in and use of online assessments across a range of disciplines has raised a number of issues relating to student perceptions and performance when using this medium. The validity of such online tests is a crucial consideration, especially if they are to be used for summative as well as formative assessment. The current study examined higher education students' performance on online assessments which they were required to take as part of an undergraduate psychology course. The marks obtained by students required to take the same multiple choice question (MCQ) assessment online and offline (in pen-and-paper format) were compared, and relationships between performance and computer anxiety and computer engagement measures were explored. The results indicate minimal influence of assessment modality, computer anxiety and computer engagement on MCQ test scores, with only very small and non significant effect sizes being observed overall. No evidence of any significant relationships between gender and age and computer attitudes was observed. To conclude, the results provide promising initial support for the use of online summative assessments in contexts similar to the one used here, allaying some concerns about disadvantaging certain groups of students. However, further research is needed to explore possible performance differences across different contexts, assessment types, and student cohorts.

An investigation of the validity of course‐based online assessment methods: The role of computer‐related attitudes and assessment mode preferences

“…One notable exception directly addressing this question using an experimental design is presented by Hewson et al (2007). They pseudo-randomly (on the basis of surname) assigned undergraduate psychology students to take a low stake summative multiple choice question (MCQ) assignment either in online or pen and paper modes (both unproctored).…”

“…However, the proctored nature of the assessments should be noted here, whereas the present study is particularly concerned with unproctored online contexts, such as was used by Hewson et al (2007), which have been understudied to date. These authors concluded that online assessment does not seem to confer any overall disadvantage to students.…”

“…Particularly, further research is needed, which considers modality effects in relation to unproctored (online/offline) assessment tasks and implements tighter control over possible confounds with modality (following the example of Hewson et al, 2007). However, some of the findings have been equivocal and require follow-up studies.…”

“…Although students' computer-and technology-related attitudes, skills, and behaviours will no doubt have changed in various ways over recent years, this does not preclude the possibility that some students may remain disadvantaged by contexts that require computerized teaching-learning modes to be used, including e-assessments. To our knowledge, only one previous study has directly addressed this issue: Hewson et al (2007) considered relationships between computer anxiety, computer engagement, and performance on a simple, course-based, low-stake summative online MCQ assignment used to assess knowledge gained from a module delivered using traditional ftf teaching/learning methods (lectures and seminars). The drive to introduce e-assessment methods in education presupposes that candidates are competent with, and willing to engage with, ICT (JISC, 2007).…”

Web-MCQ: A set of methods and freely available open source code for administering online multiple choice question assessments

“…Preliminary support for the validity of the procedures described here has been provided by Hewson, Charlton, and Brosnan (2007). These authors compared online and offline administrations of the same (summative) MCQ assessment (using the third implementation described in the present paper), and examined whether performance in either context was related to levels of computer anxiety and/or engagement.…”