Markers’ criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

Abstract

Background: Marking of essays is mainly carried out by human raters who bring in their own subjective and idiosyncratic evaluation criteria, which sometimes lead to discrepancy. This discrepancy may in turn raise issues like reliability and fairness. The current research attempts to explore the evaluation criteria of markers on a national level high stakes examination conducted at 12th grade by three examination boards in the South of Pakistan. Methods: Fifteen markers and 30 students participated in the study. For this research, data came from quantitative as well as qualitative sources. Qualitative data came in the form of scores on a set of three essays that all the fifteen markers in the study marked. For the purpose of this study, they weren't provided with any rating scale as to replicate the current practices. Qualitative data came from semi-structured interviews with the selected markers and short written commentaries by the markers to rationalize their scores on the essays. Results: Many-facet Rasch model analyses present differences in raters' consistency of scoring and the severity they exercised. Additionally, an analysis of the interviews and the commentaries written by raters justifying the scores they gave showed that there is a great deal of variability in their assessment criteria in terms of grammar, attitude towards mistakes, handwriting, length, creativity and organization and use of cohesive devices. Conclusions: The study shows a great deal of variability amongst markers, in their actual scores as well as in the criteria they use to assess English essays. Even they apply the same evaluation criteria, markers differ in the relative weight they give them.

Practices of Examiners in Marking English Essay at HSSC Level

“…In contrast, a detailed marking sheet is provided to the paper checkers of O level with an intelligent distribution of marks, which checks the multiple elements of language. Fernandez and Siddiqui (2017) reported that raters are not provided with evaluation criteria in the form of a rubric in scoring an English paper for high stake exam conducted by BISE in Punjab. It also means bypassing the objectives behind designing the paper.…”

“…It is because the students focus more on the use of pen and markers than the content. The students with average or unclear handwriting get low scores even after writing quality content on paper, e.g., the different scorers have different expectations from a good piece of writing to their subjective opinion sometimes works as a central factor affecting the reliability of marking (Fernandez and Siddiqui, 2017). On contrary, O level marking criteria focuses on achievable objectives, i.e., awareness of the candidate about a topic, required points for development and variety of justifications with appropriate tone and register rather than considering handwriting as a basis for awarding scores.…”

“…In response to the question about the judgement of the length of an essay, two options were generally chosen 143(40.9%) paper checkers have selected the option 'from paragraphing', and 145 (41.4%) paper checkers were in view to apply their judgement. Fernandez and Siddiqui (2017) have reported that the rater's experience and the elements of the essay enable them to guess about the length of an essay. The situation again appeals to have a proper valid and reliable draft to score the English paper in promoting skills-based learning.…”

“…A majority of paper raters (50.9%) have perceived themselves as average, while 25.7% and 23.4% of paper checkers perceive themselves as lenient and strict examiners, respectively. Marking of an essay is still taken as a human job, which brings subjective and idiosyncratic evaluation and sometimes leads to discrepancy Fernandez and Siddiqui (2017), e.g., it may raise serious questions on fairness and reliability of marking criteria. The response to the question about marking the fully satisfied essay shows surprising results, i.e., 109 (31.1%) raters have recorded that they give 90% marks to a fully satisfying essay.…”

Opportunities and Questions: A Short Report on Rubric Assessments in Asia and the Middle East

“…As already discussed, the commonplace holistic bias toward overemphasis upon grammar, spelling, and other organizational and meta-linguistic features and briefly discussed teacher perspectives as a key factor which more heavily affects holistic scoring than rubric scoring (or that of other analytic measures). Fernandez and Siddiqui (2017) also point out that rater scoring differences of writing often result from the severity or leniency of their grading preferences. In their 2017 study of fifteen Pakistani writing raters, the respondents rated the same three essays holistically according to their usual scoring practices.…”