Exploring the nature of examiner thinking during the process of examination marking

Crisp, Victoria

doi:10.1080/03057640802063486

Cited by 39 publications

(36 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, these data collection approaches are imperfect. Some participants are better at thinking aloud than others and which can lead some participants to grade student work more slowly (Crisp 2008). Some participants in Department A said very little while they were assessing the work which suggests that they struggled to articulate their thinking.…”

Section: Methodological Challengesmentioning

confidence: 99%

Making judgements about students making work: Lecturers’ assessment practices in art and design

Orr

Bloxham

2012

Arts and Humanities in Higher Education

View full text Add to dashboard Cite

This document is the author deposited version. You are advised to consult the publisher's version if you wish to cite from it. Published versionORR, Susan and BLOXHAM, Sue (2013). Making judgements about students making work : lecturers' assessment practices in art and design. Arts and Humanities in Higher Education, 12 (2-3), 234-253. Copyright and re-use policy AbstractThis research study explores the assessment practices in two higher education art and design departments. The key aim of this research was to explore art and design studio assessment practices as lived by and experienced by art and design lecturers. This work draws on two bodies of pre existing research. Firstly this study adopted innovative methodological approaches that have been employed to good effect to explore assessment in text based subjects (think aloud) and moderation mark agreement (observation). Secondly the study builds on existing research into the assessment of creative practice. By applying thinking aloud methodologies into a creative practice assessment context the authors seek to illuminate the 'in practice' rather than espoused assessment approaches adopted. The analysis suggests that lecturers in the study employed three macro conceptions of quality to support the judgement process. These were; the demonstration of significant learning over time, the demonstration of effective studentship and the presentation of meaningful art/design work. Introduction: Studio based PedagogyThe studio is at the heart of art and design education in higher education. Typically undergraduate art and design students are set open-ended project briefs that direct the development of their individual art/design practice. This project centered learning approach is a defining element of studio based pedagogy. Students interpret the project brief and carry out research and development activity that culminates in the submission of a body of work. This pedagogy can be characterised as 'self directed [..] and very free form curriculum' (Svensson and Edstrom (2011, 1). At different points over the course of study there will be studio Crits. The term Crit describes studio based formative or summative feedback in small or large group situations with students and lecturers (Blythman, Blair and Orr 2008). These Crits offer an opportunity for staff and students to discuss and evaluate the work.

show abstract

Section: Methodological Challengesmentioning

confidence: 99%

Making judgements about students making work: Lecturers’ assessment practices in art and design

Orr

Bloxham

2012

Arts and Humanities in Higher Education

View full text Add to dashboard Cite

show abstract

“…The list of features was derived from our reading of the literature on examiner processes (Crisp, 2008;Pollitt & Murray, 1996;Suto & Greatorex, 2008;Suto & Nadas, 2009) as well as our scrutiny of the students' work. The judges were asked to "indicate the influence of the listed features when judging" using a five-point Likert scale from -2 ("Strong negative influence") to +2 ("Strong positive influence").…”

Section: Judging Processesmentioning

confidence: 99%

The problem of assessing problem solving: can comparative judgement help?

Jones

Inglis

2015

Educ Stud Math

View full text Add to dashboard Cite

School mathematics examination papers are typically dominated by short, structured items that fail to assess sustained reasoning or problem solving. A contributory factor to this situation is the need for student work to be marked reliably by a large number of markers of varied experience and competence. We report a study that tested an alternative approach to assessment, called comparative judgement, which may represent a superior method for assessing open-ended questions that encourage a range of unpredictable responses. An innovative problem solving examination paper was specially designed by examiners, evaluated by mathematics teachers, and administered to 750 secondary school students of varied mathematical achievement. The students' work was then assessed by mathematics education experts using comparative judgement as well as a specially designed, resourceintensive marking procedure. We report two main findings from the research. First, the examination paper writers, when freed from the traditional constraint of producing a mark scheme, designed questions that were less structured and more problem-based than is typical in current school mathematics examination papers. Second, the comparative judgement approach to assessing the student work proved successful by our measures of inter-rater reliability and validity. These findings open new avenues for how school mathematics, and indeed other areas of the curriculum, might be assessed in the future.Keywords: assessment, problem solving, validity, comparative judgement 3 Typical mathematics exams are not fit for the purpose of assessing students' mathematical knowledge and skills. Analyses of the content and style of examination papers support the conjecture that mathematics examination papers comprise mainly short items that assess the rote learning of isolated facts and procedures (Berube, 2004; NCETM, 2009; Noyes, Wake, Drake & Murphy, 2011). An example question from a recent General Certificate for Secondary Education (GCSE) examination paper, a national qualification in England taken by most school leavers, illustrates the problem, as shown in Figure 1. ***FIGURE 1 ABOUT HERE*** At first glance, the question looks promising for assessing students' mathematical knowledge and skills. It makes use of a calendar context thereby appealing to the everyday relevance of mathematics. It also builds on students' experience of a counting system grouped in 7s to introduce an interesting generality that wherever the 2 by 2 square is positioned the provided algorithm will always give 7. However, to achieve full marks, all a student needs to do is compute the provided algorithm using the provided inputs. No explanation or proof of why the result is always 7 is required or rewarded. An efficient examination taker can achieve full marks without noticing there is a mathematically interesting generality at all.The question might be improved by asking students to compute the algorithm for a few 2 by 2 squares of their own choosing, and then asking them to explain what they ...

show abstract

“…In particular, Yorke makes the point that 'assessors' grading behaviour is tacitly influenced by norm referencing and Vaughan (1991), Shay (2004), Delaney (2005) andOrrell (2008) also found that tutors draw on their knowledge of different students' work in order to make their judgements. More broadly, Crisp (2008) and found that other students' work affects individuals' standards' frameworks. There is no evidence that external examiners will not be influenced in the same way in making their judgements.…”

Section: External Examiners Can Represent Community Standards Reliablmentioning

confidence: 99%

“…Using the 'Heuristics and Biases' approach, research (Gilovich et al 2002) suggests that expert judgement in cases of uncertainty or complexity utilises a number of simplifying cues (heuristics) because a more rational and detailed judgement is too difficult or time consuming. Such heuristics allow for systematic biases to creep in of which the expert is unaware and may lead to assessors predicting performance in advance of completing reading (Crisp 2008). For example, recent research has identified that knowledge of the student, group stereotyping and surface features of work can be sources of bias (Brooks 2012;Kangis 2001;Hartley et al 2006).…”

mentioning

confidence: 99%

External examining: fit for purpose?

Bloxham

Price

2013

Studies in Higher Education

View full text Add to dashboard Cite

Usage of any items from the University of Cumbria's institutional repository 'Insight' must conform to the following fair usage guidelines.Any item and its associated metadata held in the University of Cumbria's institutional repository Insight (unless stated otherwise on the metadata record) may be copied, displayed or performed, and stored in line with the JISC fair dealing guidelines (available here) for educational and not-for-profit activities provided that• the authors, title and full bibliographic details of the item are cited clearly when any part of the work is referred to verbally or in the written form• a hyperlink/URL to the original Insight record of that item is included in any citations of the work • the content is not changed in any way• all files required for usage of the item are kept together with the main item file. You may not• sell any part of an item• refer to any part of an item without citation • amend any item or contextualise it in a way that will impugn the creator's reputation• remove or alter the copyright statement on an item. AbstractIn a context of international concern about academic standards, the practice of external examining is widely admired for its role in defending standards. Yet a contradiction exists between this faith in examining and continuing concerns about standards. This article argues that external examining rests on assumptions about standards which are significantly open to challenge. Six assumptions relating to the conceptual context, the operation and the nature of examiners themselves are analysed drawing on a review of the available evidence. The analysis challenges the notion of a consensus on standards and the potential to vest in individuals the ability to represent that consensus when judging the comparability of academic standards in a stable and appropriate way.The issues raised have relevance to the UK and to other national systems using external examiners or seeking to guarantee academic standards by, in some cases, adopting quality assurance approaches developed in the UK. 2 External examining: fit for purpose?External examining is a process by which the fairness of assessment and the academic standards in higher education are monitored. This monitoring normally involves an academic from another institution approving assessment tasks and examinations and checking a sample of student work for the courses which they examine. A wider range of activities may also be included as discussed below. In certain limited circumstances, for example in viva voces for research degrees, external examiners directly examine student performance. In the case of vocational courses, some examiners may be practising members of the relevant profession rather than academics. External examining is prevalent in the UK and several other countries operate or have operated an external examiner system:

show abstract

Exploring the nature of examiner thinking during the process of examination marking

Cited by 39 publications

References 20 publications

Making judgements about students making work: Lecturers’ assessment practices in art and design

Making judgements about students making work: Lecturers’ assessment practices in art and design

The problem of assessing problem solving: can comparative judgement help?

External examining: fit for purpose?

Contact Info

Product

Resources

About