A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas

Abstract

Onscreen marking (OSM) has been used for the majority of Hong Kong public examinations since 2012. The current study compares marker reactions to OSM, ie, perceived ease of use and acceptance of OSM, against the backdrop of virtually all subject areas being marked on screen. The data were collected from three major sources: (1) survey data obtained from 1743 markers across 14 major subject areas, (2) markers' qualitative comments about the OSM system and (3) post-hoc interviews with a key informant from the Hong Kong Examinations and Assessment Authority (HKEAA). Results showed that, in general, markers revealed a high level of perceived ease of use and positive acceptance of OSM. The effect of subject area for both scales was statistically significant. On the Ease of Use in the OSM Environment scale, markers of information and communication technology (ICT) and mathematics were the most positive, with markers of history and geography the least positive. On the Acceptance of OSM scale, markers of mathematics and ICT were the most accepting, with markers of biology and geography the least. The analysis of survey data was triangulated by markers' qualitative comments together with the HKEAA staff interview. Possible explanations for the results are proposed and implications for the further development of OSM are briefly discussed.In the UK, Cambridge Assessment (as it is now known) first experimented with the OSM of scanned paper scripts in 1999. After further piloting in 2006, Cambridge Assessment invested substantial sums into OSM for the 5-year period up to 2012 (Raikes, Greatorex & Shaw, 2004). RM Results (2014), an organisation specialising in onscreen testing and OSM, claims that 143 million exam pages were scanned and marked on screen in 2013 using RM Assessor, an OSM application. The official body of the UK Government Ofqual (the Office of Qualifications and Examinations Regulation) has declared that "onscreen marking is now the main type of marking

The Impact of Computers on Marking Behaviors and Assessment: A Many-Facet Rasch Measurement Analysis of Essays by EFL College Students

“…It is worth mentioning that several studies conducted in the United Kingdom and Hong Kong have lent substantial support to comparability (Coniam & Yan, 2016;Johnson et al, 2012;Johnson, Nádas, & Bell, 2010;Raikes, Greatorex, & Shaw, 2004). In these studies, the rater scores awarded to scanned scripts and paper originals were analyzed and contrasted by computing interrater indexes such as exact agreement indexes, Cohen's κ, Pearson's r, Kendall's τ−b, or intraclass correlation coefficients (ICCs).…”

Assessment of ICT in Tertiary Education Applying Structural Equation Modeling and Rasch Model

“…In such a case, we assumed that combining these two analytical tools would provide a great opportunity for conducting meaningful measurements, which could potentially be generalized and used in other situations. Coniam and Yan (2016) claimed that the widespread development of ICT has had progressively powerful effects on numerous aspects of education, including assessment. Hence, it is assumed here that Chinese university lecturers' ICT skills would be valuable for performing their measurement activities, if they frequently used ICT for teaching, research, and learning purposes.…”