Improving marking reliability of scientific writing with the Developing Understanding of Assessment for Learning programme

Abstract

The Developing Understanding of Assessment for Learning (DUAL) programme was developed with the dual aims of improving both the quality and consistency of feedback students receive and the students' ability to use that feedback to improve. DUAL comprises a range of processes (including marking rubrics, sample reports, moderation discussions and peer-review activities) which support explicit knowledge exchange between staff, and between staff and students and the development of tacit knowledge of standards in all participants. This paper describes the study which quantified the extent to which the DUAL programme improved reliability and efficiency of marking of first year biology reports by a large team of laboratory demonstrators. Marker reliability increased after participation in DUAL, with demonstrators marking laboratory reports significantly closer to the marks awarded by an expert marker of the discipline. Efficiency was also improved since time taken to mark the reports did not increase significantly, despite the fact that demonstrators were adapting to a new system of marking. Demonstrators valued the programme and nominated the marking rubric, feedback code and moderation discussion as the most useful elements. The community of practice built through joint participation in the DUAL programme created a supportive working environment for demonstrators and strengthened the collaboration between the discipline and academic language and learning lecturers.

Feedback codes and action plans: building the capacity of first-year students to apply feedback to a scientific report

“…The 'fair' dimension of feedback focuses on clarity and consistency of the message, which was one of the reasons for the development of the feedback code (Bird and Yucel 2013). The code is used by markers to annotate reports, and the codes are explained in a resource made available to all students, thus ensuring the consistency of the message.…”

“…Marking inconsistency, particularly in a large teaching team, results from differences between markers as well as differences in the way individual markers assess each assessment piece (Bloxham 2009). Marking inconsistencies can be minimised with well-developed marking resources, including rubrics which make the assessment criteria and standards explicit (Rust, Price, and O'Donovan 2003), and moderation discussions, where explicit knowledge about criteria and standards is exchanged and tacit knowledge of standards is developed in each participant (Bird and Yucel 2013). Consistency in provision of feedback can also be gained by developing global statements relevant to multiple students (Vardi 2009), and providing markers with a bank of sample comments or codes (Bird and Yucel 2013).…”

“…The DUAL programme includes a staged training programme for the team of laboratory demonstrators (Bird and Yucel 2013), alongside a series of formative and summative classroom activities which aim to develop expertise in students over a 13-week semester ( Figure 1). The majority of the cohort is commencing first-year students, and this programme plays an important role in supporting students to learn how to effectively use feedback.…”

The Benefits of Peer Review and a Multisemester Capstone Writing Series on Inquiry and Analysis Skills in an Undergraduate Thesis

“…To achieve these benefits, however, students must be taught the characteristics of effective science writing and how to read effectively for offering useful feedback (Morgan et al , 2011). Lu and Law (2012) and Bird and Yucel (2013) stress the importance of instructing students to see how effective peer review goes beyond the ability to recognize substandard work and instead helps writers advance higher-order aspects of their writing such as logic, coherence, and evidence selection (Glaser, 2014). By thinking beyond “error” as peer reviewers, students are able to gain insight on their own science reasoning and analysis (Morgan et al ., 2011; Glaser, 2014).…”

“…To this end, practitioners and scholars highlight the need for science writing rubrics in classes that integrate writing (Reynolds and Thompson, 2011; Bird and Yucel, 2013; Mynlieff et al , 2014; Dowd et al , 2015b). Science writing rubrics articulate for instructors and students the aspects of a writing project most salient to science reasoning—clear research focus, well-reasoned data analysis, logically drawn implications for future research, and so on—while deprioritizing surface-level errors (Morgan et al , 2011; Glaser, 2014).…”