Who should mark what? A study of factors affecting marking accuracy in a biology examination

Suto, Irenka; Nádas, Rita; Bell, James P.

doi:10.1080/02671520902721837

Cited by 19 publications

(21 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(The IGCSE is a qualification for 14-to 16-year-olds, taken in over a hundred countries, and in at least 50 subjects.) As described in full by Suto, Nádas, and Bell (2009), 23 diverse biology questions were explored, varying along several dimensions, including format, number of marks, difficulty for candidates and mark scheme complexity. For each question, stratified sampling methods were used to draw representative samples of past candidates' responses.…”

Section: Methodsmentioning

confidence: 99%

“…In addition to those affecting personal marking skills, such as education and experience, some factors, such as question format and response clarity, could affect task demands (Suto and Nádas 2008). As part of a wider empirical study investigating factors contributing to marking accuracy for an international biology examination (Suto, Nádas, and Bell 2009), we explored the question of whether marking speed is also related to accuracy. This is important for training examiners and planning realistic marking schedules.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speed isn’t everything: a study of examination marking

Nádas¹,

Suto²

2010

Educational Studies

Self Cite

View full text Add to dashboard Cite

The question of whether marking speed is related to marking accuracy is important for training examiners and planning realistic marking schedules. We explored marking speed in the context of a past examination for an international biology qualification for 14-to 16-year-olds. Forty-two markers with differing backgrounds experimentally marked 23 diverse examination questions. All responded to questionnaires about times taken to mark two of four samples of candidate responses. We demonstrated a positive practice effect for inexperienced markers, who became significantly faster during the course of their marking whilst maintaining their accuracies; there was no clear trade-off between speed and accuracy. The benefits of marking practice and background experience are distinct phenomena. To improve accuracy, longer term investments in education and experience are needed. BackgroundWhen marking examinations, examiners often wonder how long to spend on each student's work. Although time pressures can be immense, it might reasonably be assumed that, as with other cognitively demanding tasks such as complex mental arithmetic or piano-playing, there is a trade-off between marking speed and accuracy. If this trade-off exists, then optimum marking speed may depend on each examiner's personal expertise. Practice effects are established in many fields, and theorists such as Anderson (1982) and Dreyfus and Dreyfus (1986) have argued that as expertise develops, some tasks can be completed more rapidly and automatically. This is because procedures become standardised and routine, and decision-making becomes less laboured and more intuitive.The process of marking examinations is often complex, with many factors potentially at play. In addition to those affecting personal marking skills, such as education and experience, some factors, such as question format and response clarity, could affect task demands (Suto and Nádas 2008). As part of a wider empirical study investigating factors contributing to marking accuracy for an international biology examination (Suto, Nádas, and Bell 2009), we explored the question of whether marking speed is also related to accuracy. This is important for training examiners and planning realistic marking schedules.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Speed isn’t everything: a study of examination marking

Nádas¹,

Suto²

2010

Educational Studies

Self Cite

View full text Add to dashboard Cite

show abstract

“…Pinot de Moira [2011] suggested involving many markers per script as one way of achieving marking reliability while Suto et al [2011] were of the opinion that by giving complex questions to be marked by seasoned examiners ensures reliability in the marking process. Research by Tisi et al [2013] and Pinot de Moira, 2011] have revealed several ways in which Item level marking lead into attainment of marking reliability.…”

Section: Marking Reliabilitymentioning

confidence: 99%

“…The Supervisor can adjust the allocation of the question if there is a need. Difficult questions can be allocated to seasoned markers while the less experienced markers are assigned easy questions [Suto et al, 2011].…”

Section: Marking Systemsmentioning

confidence: 99%

“…These Markers can also be allocated questions which were answered by many Candidates. Research carried by Pinot de Moira [2011]; Meadows and Billington [2007]; Suto et al [2011] has shown that mark reliability in item level marking can be achieved by allocating complex questions to more experienced markers and simple questions to junior markers. 2).Roving Markers: These were regarded as fast and accurate Markers who can mark their allocated question(s) and go on to assist in marking questions assigned to them by the Belt Marking Supervisor.…”

Section: Challenges Of Introducing Belt Marking At Great Zimbabwe Unimentioning

confidence: 99%

See 1 more Smart Citation

Traditional or Conveyor belt marking: Exploring the way forward at Great Zimbabwe University, Zimbabwe

Risiro¹

2014

GJER

View full text Add to dashboard Cite

The purpose of the study was to examine the strengths and weaknesses of the current traditional marking and explore possibilities and challenges of introducing conveyor belt marking at Great Zimbabwe University. Individual interviews were carried out for students enrolled in the Bachelor of Education Programmes. Lecturers who have been involved in conveyor belt marking were purposefully sampled to answer questionnaires. There are possibilities of introducing belt marking at the Institution. Students were more supportive of using belt marking unlike the Lecturers who pointed out too many challenges before implementing it. The challenges experienced in belt marking were organisational and management of the marking process. The solutions to the challenges involved identifying subject specialists across the University departments, staff recruitment and in service training for Staff. Departmental Chairpersons and Subject Coordinators across Faculties need to liaise with each other for the smooth running of the marking.

show abstract

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

Who should mark what? A study of factors affecting marking accuracy in a biology examination

Cited by 19 publications

References 17 publications

Speed isn’t everything: a study of examination marking

Speed isn’t everything: a study of examination marking

Traditional or Conveyor belt marking: Exploring the way forward at Great Zimbabwe University, Zimbabwe

A Historical Survey of Research Regarding Constructed-Response Formats

Contact Info

Product

Resources

About