Judging Text Presented on Screen: Implications for Validity

Johnson, Martin; Greatorex, Jackie

doi:10.2304/elea.2008.5.1.40

Cited by 5 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Johnson and Greatorex (2008) argue that this issue might also affect those assessing longer texts on screen, implying a need to carry out studies exploring how judgements about longer textual performances might be influenced when assessors read them in different modes. This concern resonates with another recent screen marking study, which found correlations between re-marked essays to be significantly lower when those scripts were re-marked on screen compared with their re-mark on paper (Fowles, 2008).…”

Section: British Journal Of Educational Technologymentioning

confidence: 99%

Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Johnson¹,

Nádas²,

Bell³

2010

Brit J Educational Tech

Self Cite

View full text Add to dashboard Cite

There is a growing body of research literature that considers how the mode of assessment, either computer-based or paper-based, might affect candidates' performances. Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor consistency when dealing with extended textual answers in different modes. This research project explored whether the mode in which a set of extended essay texts were accessed and read systematically influenced the assessment judgements made about them. During the project, 12 experienced English literature assessors marked two matched samples of 90 essay exam scripts on screen and on paper. A variety of statistical methods were used to compare the reliability of the essay marks given by the assessors across modes. It was found that mode did not present a systematic influence on marking reliability. The analyses also compared examiners' marks with a gold standard mark for each essay and found no shifts in the location of the standard of recognised attainment across modes. IntroductionLiterature suggests that the feasibility, validity and reliability of working on screen have long been the focus of audiences from a wide range of backgrounds, ranging from studies in the contexts of education to those in occupational and cognitive psychology, among others. This particular study sought to investigate whether the mode of marking (on screen or paper) had any influence on essay marking reliability and markers' leniency/rigour.

show abstract

Section: British Journal Of Educational Technologymentioning

confidence: 99%

Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Johnson¹,

Nádas²,

Bell³

2010

Brit J Educational Tech

Self Cite

View full text Add to dashboard Cite

show abstract

“…The challenges relate mostly to the usability of the systems, e.g. judgements are impaired when moderators must manage different pages in documents [10]. Searching through different portfolios presents difficulties when viewing and comparing different candidates' work.…”

Section: Benefits and Challenges Of Emoderationmentioning

confidence: 99%

Requirements for an eModeration System in Private Schools in South Africa

Rajamany¹,

Staden²,

Biljon³

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Despite the growing importance of digitization in all aspects of teaching and learning, digital moderation (eModeration) has received little attention in research or practice. Considering the secondary school environment, no evidence-based requirements could be found for the development of a digital moderation system. This finding provides the rationale for this investigation into the requirements for an efficient eModeration system for IT and CAT assessments at grade 12 level in South Africa (SA). This study draws on eModeration literature as well as technology adoption literature to identify a set of preadoption eModeration requirements against the background of the challenges and benefits of eModeration. The research design involved a single exploratory case study with IT and CAT moderators as units of analysis. Qualitative data analysis using Atlas.ti V8 was conducted on the 61 responses received from an online survey and 4 responses received from a focus group interview. The findings confirmed the absence of customized eModeration systems. The main contribution is the eModeration requirements identified from literature and then refined by triangulation with the empirical findings of the survey. The research has practical value in guiding the design of eModeration systems for the school environment.

show abstract

“…This article contributes to a growing research literature which explores how the technological environment (mode) in which marking is carried out influences marking processes (Price & Petre, 1997;Whetton & Newton, 2002;Royal-Dawson, 2003;Sturman & Kispal, 2003;Fowles & Adams, 2005;Fowles, 2008;Johnson & Greatorex, 2008;Coniam, 2009Coniam, , 2010Coniam, , 2011Johnson et al, 2009). Screen and paper-based marking modes each possess characteristics or 'affordances' (Gibson, 1979), and it is possible that these can affect both the manual and the cognitive processes involved in marking a text.…”

Section: Introductionmentioning

confidence: 99%

“…Johnson and Greatorex (2008) point out that there has been more consideration about how the reading of longer texts on screen affects students being assessed than there has about those who are responsible for marking such texts, although the issues for both groups should be synonymous. Very recently this balance has been slightly redressed by a number of studies (Fowles, 2008;Shaw & Imam, 2008;Coniam, 2009;Johnson et al, 2009).…”

Section: Introductionmentioning

confidence: 99%

Marking Extended Essays on Screen: Exploring the Link between Marking Processes and Comprehension

Johnson¹,

Hopkin²,

Shiell³

2012

E-Learning and Digital Media

Self Cite

View full text Add to dashboard Cite

Technological developments are impacting upon UK assessment practices in many ways. For qualification awarding bodies, a key example of such impact is the ongoing shift towards examiners marking digitally scanned copies of examination scripts on screen rather than the original paper documents. This digitisation process has obvious benefits, affording flexible information management and distribution. Digitisation can also lead readers to interact differently with texts, with readers' physical engagement with digital texts differing from traditional paper-based texts. More specifically, it appears that reader navigation and annotation behaviours are particularly influenced by a digital mode shift. A consequence of this is that it is possible that the mental workload involved in reading on screen is greater than when reading on paper. Moreover, it is theorised that this additional load can have an adverse affect on reader comprehension building. These findings prompt questions about whether the mode of marking might unduly influence marking outcomes when assessors read longer essay responses in a digital environment. In light of this concern, this article explores how marking mode influences examiner essay marking behaviours, which in turn might influence their comprehension building and their marking outcomes. This study gathers data from 12 experienced examiners working within a large UK-based awarding body to investigate how mode influences their navigation and annotation behaviours when marking extended essays on paper and on screen. These comparative data are then used to explore a model which suggests that manual marking behaviours, such as navigation and annotation, can influence examiners' comprehension whilst marking. The implications for future developments in digital essay-marking software are then discussed.

show abstract

Judging Text Presented on Screen: Implications for Validity

Cited by 5 publications

References 14 publications

Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Marking essays on screen: An investigation into the reliability of marking extended subjective texts

Requirements for an eModeration System in Private Schools in South Africa

Marking Extended Essays on Screen: Exploring the Link between Marking Processes and Comprehension

Contact Info

Product

Resources

About