Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Johnson, Martin; Nádas, Rita; Bell, James P.

doi:10.1111/j.1467-8535.2009.00979.x

Cited by 28 publications

(22 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The quantitative analysis of the original study investigating possible mode-related differences in marking reliability found that the mode of assessment did not have a statistically significant influence on examiners' accuracy (Johnson, Nádas, and Bell, 2009). The analysis reported in this paper is based on additional qualitative investigations which were used to augment these quantitative findings since mixed research methods are well suited to shed more light on the judgement processes involved in assessing in different modes.…”

Section: Methodsmentioning

confidence: 98%

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension

Johnson¹,

Nádas²

2009

Learning, Media and Technology

Self Cite

View full text Add to dashboard Cite

Within large scale educational assessment agencies in the UK, there has been a shift towards assessors marking digitally scanned copies rather than the original paper scripts that were traditionally used. This project uses extended essay examination scripts to consider whether the mode in which an essay is read potentially influences the judgements made about it, employing qualitative and quantitative data-gathering approaches to focus on the annotation practices of readers who are assessing extended essays, and to explore whether this might be affected by mode. The project also gathers evidence about the spatial encoding ability of these readers to suggest that mode-related influences on annotation might influence the ability of readers to comprehend fully the material being read. Examiners had less spatial awareness about the location of script features on screen than on paper; however, annotating on screen might link to both the provision of appropriate annotations and the relative ease with which these are deployed whilst reading. There was also evidence that on-screen annotation could contribute to the development of good mental representations of texts on screen. IntroductionThere is a growing body of research literature that considers how the mode of assessment, either computer-or paper-based, might affect students' performances (Paek 2005). Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor behaviour when dealing with extended written essay answers in different modes. This might be considered surprising since research literature from domains such as ergonomics, human factors, human-computer interaction, and the psychology of reading suggests that the mode in which longer texts are read might be expected to influence the way that readers access and comprehend such texts (324 M. Johnson and R. Nádas ability of these readers to suggest that mode-related influences on annotation might influence the ability of readers to fully comprehend the material being read. This issue has particular relevance for the domain of educational assessment where there are moves towards increased digital script marking, as annotation has a clear formal quality assurance function within large scale assessment systems. What is less clearly articulated is the informal utility of annotations for assessors whilst comprehension building, a function that carries implications beyond assessment and into the realms of human cognition, cognitive load and the psychology of reading. BackgroundBennett (2002) describes the rapid growth of computer technology use in workplaces and education as inexorable. Although technology offers the potential to broaden educational assessment beyond what traditional methods allow, there are inevitable concerns during a transition phase (where assessments exist in both paper-and computer-based modes) that their outcomes are not comparable. In her review of comparability studies since 1993, Paek (2005 ...

show abstract

Section: Methodsmentioning

confidence: 98%

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension

Johnson¹,

Nádas²

2009

Learning, Media and Technology

Self Cite

View full text Add to dashboard Cite

show abstract

“…The study found that there was no substantial evidence to indicate that extended-essay-marking accuracy was influenced by marking mode. The study also replicated the earlier Johnson et al (2009) study by gathering additional evidence of the marking processes used by examiners whilst marking. This article looks at these additional data to consider how the examiners reached those similar outcomes.…”

Section: Introductionmentioning

confidence: 81%

“…Johnson and Greatorex (2008) point out that there has been more consideration about how the reading of longer texts on screen affects students being assessed than there has about those who are responsible for marking such texts, although the issues for both groups should be synonymous. Very recently this balance has been slightly redressed by a number of studies (Fowles, 2008;Shaw & Imam, 2008;Coniam, 2009;Johnson et al, 2009). These studies have compared marking accuracy levels on screen and on paper using essays with an approximate maximum of 600 words.…”

Section: Introductionmentioning

confidence: 95%

“…This article contributes to a growing research literature which explores how the technological environment (mode) in which marking is carried out influences marking processes (Price & Petre, 1997;Whetton & Newton, 2002;Royal-Dawson, 2003;Sturman & Kispal, 2003;Fowles & Adams, 2005;Fowles, 2008;Johnson & Greatorex, 2008;Coniam, 2009Coniam, , 2010Coniam, , 2011Johnson et al, 2009). Screen and paper-based marking modes each possess characteristics or 'affordances' (Gibson, 1979), and it is possible that these can affect both the manual and the cognitive processes involved in marking a text.…”

Section: Introductionmentioning

confidence: 97%

“…In addition to the marking accuracy outcomes reported in Johnson et al (2009), further analyses were carried out as part of that study to explore the impact of marking mode on examiners' manual marking processes. These additional analyses were reported in two articles which suggested that, despite reaching equivalent marking outcomes, examiners' navigation and annotation behaviours were influenced by marking mode (Johnson & Nádas 2009a,b).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Marking Extended Essays on Screen: Exploring the Link between Marking Processes and Comprehension

Johnson¹,

Hopkin²,

Shiell³

2012

E-Learning and Digital Media

Self Cite

View full text Add to dashboard Cite

Technological developments are impacting upon UK assessment practices in many ways. For qualification awarding bodies, a key example of such impact is the ongoing shift towards examiners marking digitally scanned copies of examination scripts on screen rather than the original paper documents. This digitisation process has obvious benefits, affording flexible information management and distribution. Digitisation can also lead readers to interact differently with texts, with readers' physical engagement with digital texts differing from traditional paper-based texts. More specifically, it appears that reader navigation and annotation behaviours are particularly influenced by a digital mode shift. A consequence of this is that it is possible that the mental workload involved in reading on screen is greater than when reading on paper. Moreover, it is theorised that this additional load can have an adverse affect on reader comprehension building. These findings prompt questions about whether the mode of marking might unduly influence marking outcomes when assessors read longer essay responses in a digital environment. In light of this concern, this article explores how marking mode influences examiner essay marking behaviours, which in turn might influence their comprehension building and their marking outcomes. This study gathers data from 12 experienced examiners working within a large UK-based awarding body to investigate how mode influences their navigation and annotation behaviours when marking extended essays on paper and on screen. These comparative data are then used to explore a model which suggests that manual marking behaviours, such as navigation and annotation, can influence examiners' comprehension whilst marking. The implications for future developments in digital essay-marking software are then discussed.

show abstract

The Final Validation Study: A Comparative Picture of the Ease of Use and Acceptance of Onscreen Marking by Markers Across Subject Areas

Coniam

Falvey

Yan

2016

Validating Technological Innovation

View full text Add to dashboard Cite

Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Cited by 28 publications

References 19 publications

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension

Marking Extended Essays on Screen: Exploring the Link between Marking Processes and Comprehension

The Final Validation Study: A Comparative Picture of the Ease of Use and Acceptance of Onscreen Marking by Markers Across Subject Areas

Contact Info

Product

Resources

About