Extended essay marking on screen: is examiner marking accuracy influenced by marking mode?

Johnson, Martin; Hopkin, Rebecca; Shiell, Hannah; Bell, James P.

doi:10.1080/13803611.2012.659932

Cited by 7 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, reading on screen was found to be a more cognitively demanding task than reading on paper (Wästlund, Reinikka, Norlander & Archer, ) and such increased cognitive workload might result in lower levels of comprehension of a text (Mayes, Sims & Koonce, ). This effect was most significant when readers read longer texts than when they read short texts (Johnson et al , ).…”

Section: Resultsmentioning

confidence: 99%

“…Accompanying the wide use of OSM is the increasing research attention on relevant areas including the comparability between paper‐based marking and OSM (eg, Geranpayeh, ; Johnson, Hopkin, Shiell & Bell, ; Johnson, Nádas & Bell, ), the development of OSM systems (eg, Campbell, ; Ramakrishna, Navya Sree, Sri Harish, Swarna & Vasundhara, ), as well as markers' attitudes towards OSM (eg, Coniam, , ; Yan & Coniam, ). Nevertheless, the literature about marking issues in the new assessment environment remains limited.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas

Coniam¹,

Yan²

2015

Brit J Educational Tech

View full text Add to dashboard Cite

Onscreen marking (OSM) has been used for the majority of Hong Kong public examinations since 2012. The current study compares marker reactions to OSM, ie, perceived ease of use and acceptance of OSM, against the backdrop of virtually all subject areas being marked on screen. The data were collected from three major sources: (1) survey data obtained from 1743 markers across 14 major subject areas, (2) markers' qualitative comments about the OSM system and (3) post-hoc interviews with a key informant from the Hong Kong Examinations and Assessment Authority (HKEAA). Results showed that, in general, markers revealed a high level of perceived ease of use and positive acceptance of OSM. The effect of subject area for both scales was statistically significant. On the Ease of Use in the OSM Environment scale, markers of information and communication technology (ICT) and mathematics were the most positive, with markers of history and geography the least positive. On the Acceptance of OSM scale, markers of mathematics and ICT were the most accepting, with markers of biology and geography the least. The analysis of survey data was triangulated by markers' qualitative comments together with the HKEAA staff interview. Possible explanations for the results are proposed and implications for the further development of OSM are briefly discussed.In the UK, Cambridge Assessment (as it is now known) first experimented with the OSM of scanned paper scripts in 1999. After further piloting in 2006, Cambridge Assessment invested substantial sums into OSM for the 5-year period up to 2012 (Raikes, Greatorex & Shaw, 2004). RM Results (2014), an organisation specialising in onscreen testing and OSM, claims that 143 million exam pages were scanned and marked on screen in 2013 using RM Assessor, an OSM application. The official body of the UK Government Ofqual (the Office of Qualifications and Examinations Regulation) has declared that "onscreen marking is now the main type of marking

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas

Coniam¹,

Yan²

2015

Brit J Educational Tech

View full text Add to dashboard Cite

show abstract

“…It is worth mentioning that several studies conducted in the United Kingdom and Hong Kong have lent substantial support to comparability (Coniam & Yan, 2016;Johnson et al, 2012;Johnson, Nádas, & Bell, 2010;Raikes, Greatorex, & Shaw, 2004). In these studies, the rater scores awarded to scanned scripts and paper originals were analyzed and contrasted by computing interrater indexes such as exact agreement indexes, Cohen's κ, Pearson's r, Kendall's τ−b, or intraclass correlation coefficients (ICCs).…”

Section: Comparability Of Oss Mode and Pbs Modementioning

confidence: 91%

“…Other lines of research have investigated the effects of annotations such as commenting, circling, and underlining on raters' marking results (Johnson, Hopkin, Shiell, & Bell, 2012;Johnson & Shaw, 2008;Shaw, 2008). Crisp and Johnson (2007) recruited 12 experienced raters to mark scripts written for the General Certificate of Secondary Education (GCSE) Mathematics and Business Studies qualifications in the United Kingdom.…”

Section: Comparability Of Oss Mode and Pbs Modementioning

confidence: 99%

The Impact of Computers on Marking Behaviors and Assessment: A Many-Facet Rasch Measurement Analysis of Essays by EFL College Students

2019

Sage Open

View full text Add to dashboard Cite

This study employed a mixed-design approach and the Many-Facet Rasch Measurement (MFRM) framework to investigate whether rater bias occurred between the onscreen scoring (OSS) mode and the paper-based scoring (PBS) mode. Nine human raters analytically marked scanned scripts and paper scripts using a six-category (i.e., six-criterion) rating rubric. Interviews with these raters were then conducted to gather their reflections concerning the marking experiences in the two modes. Software program FACETS was employed to estimate raters' scores, with the results indicating that (a) four raters marked the scanned scripts more severely than they did the paper scripts, whereas the remaining four raters exhibited an opposite tendency; (b) the only rater whose composite scores were comparable between the two modes still awarded harsher scores to the category of Mechanics in PBS mode; and (c) the category of Mechanics was scored more harshly in OSS mode, whereas Topic Development/Support was scored more severely in PBS mode. Analyses of quantitative and qualitative data indicated that computers affected raters' marking behaviors and their assessment, assuming a role in causing scoring bias. Implications for rater training were provided.

show abstract

“…Human raters evaluate language samples, refer to scale descriptors and apply judgement and experience to assign a final score, but there is often no quantifiable way of measuring how they weight and combine the various pieces of information in an essay. In reality, we rely on a lot of experiential judgements and knowledge on the part of examiners (Newstead and Dennis, 1994;Greatorex and Bell, 2008;Johnson et al, 2012). It is hard to explain that you know what an assessment is worth when judged against some criteria, but there is definitely an element of 'gut feeling' in marking.…”

Section: Automated Scoringmentioning

confidence: 99%

Rise of the machines? The evolving role of AI technologies in high-stakes assessment

Richardson¹,

Clesham

2021

London Review of Education

View full text Add to dashboard Cite

Our world has been transformed by technologies incorporating artificial intelligence (AI) within mass communication, employment, entertainment and many other aspects of our daily lives. However, within the domain of education, it seems that our ways of working and, particularly, assessing have hardly changed at all. We continue to prize examinations and summative testing as the most reliable way to assess educational achievements, and we continue to rely on paper-based test delivery as our modus operandi. Inertia, tradition and aversion to perceived risk have resulted in a lack of innovation (James, 2006), particularly so in the area of high-stakes assessment. The summer of 2020 brought this deficit into very sharp focus with the A-level debacle in England, where grades were awarded, challenged, rescinded and reset. These events are potentially catastrophic in terms of how we trust national examinations, and the problems arise from using just one way to define academic success and one way to operationalize that approach to assessment. While sophisticated digital learning platforms, multimedia technologies and wireless communication are transforming what, when and how learning can take place, transformation in national and international assessment thinking and practice trails behind. In this article, we present some of the current research and advances in AI and how these can be applied to the context of high-stakes assessment. Our discussion focuses not on the question of whether we should be using technologies, but on how we can use them effectively to better support practice. An example from one testing agency in England using a globally popular test of English that assesses oral, aural, reading and written skills is described to explain and propose just how well new technologies can augment assessment theory and practice.

show abstract

Extended essay marking on screen: is examiner marking accuracy influenced by marking mode?

Cited by 7 publications

References 17 publications

A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas

A comparative picture of the ease of use and acceptance of onscreen marking by markers across subject areas

The Impact of Computers on Marking Behaviors and Assessment: A Many-Facet Rasch Measurement Analysis of Essays by EFL College Students

Rise of the machines? The evolving role of AI technologies in high-stakes assessment

Contact Info

Product

Resources

About