Using verbal reports to explore rater perceptual processes in scoring: a mixed methods application to oral communication assessment

Joe, Jilliam N.; Harmes, J. Christine; Hickerson, Corey

doi:10.1080/0969594x.2011.577408

Cited by 19 publications

(17 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Rater cognition in the assessment of constructed responses has been explored by researchers in a range of contexts (e.g., Crisp, 2008a, 2010a; Greatorex & Suto, 2006; Joe, Harmes, & Hickerson, 2011; Milanovic, Saville, & Shuhong, 1996; Pula & Huot, 1993; Sanderson, 2001; Vaughan, 1991; Wolfe, 1997), but few have investigated large pieces of work such as project work or researched high‐stakes, summative assessment by classroom teachers. This research, conducted in the context of General Certificate of Secondary Education (GCSE) coursework in the United Kingdom, was designed to address this omission.…”

mentioning

confidence: 99%

An Investigation of Rater Cognition in the Assessment of Projects

Crisp¹

2012

Educational Measurement

View full text Add to dashboard Cite

In the United Kingdom, the majority of national assessments involve human raters. The processes by which raters determine the scores to award are central to the assessment process and affect the extent to which valid inferences can be made from assessment outcomes. Thus, understanding rater cognition has become a growing area of research in the United Kingdom. This study investigated rater cognition in the context of the assessment of school‐based project work for high‐stakes purposes. Thirteen teachers across three subjects were asked to “think aloud” whilst scoring example projects. Teachers also completed an internal standardization exercise. Nine professional raters across the same three subjects standardized a set of project scores whilst thinking aloud. The behaviors and features attended to were coded. The data provided insights into aspects of rater cognition such as reading strategies, emotional and social influences, evaluations of features of student work (which aligned with scoring criteria), and how overall judgments are reached. The findings can be related to existing theories of judgment. Based on the evidence collected, the cognition of teacher raters did not appear to be substantially different from that of professional raters.

show abstract

mentioning

confidence: 99%

An Investigation of Rater Cognition in the Assessment of Projects

Crisp¹

2012

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Bu alanda yapılan çalışmaların az olma sebebi çoğunlukla not verenlerin not verme esnasında karar verme süreçlerini gözlemleme imkânının olmamasıdır. Bu sebeple, notlandıranların notlandırma esnasında ne düşündüğünü ortaya çıkarmak adına sesli-düşünme protokollerinin kullanıldığı çalışmaların azlığı (örn., Joe, 2008;Joe ve diğerleri, 2011;Orr, 2002) son zamanlarda literatürde önemle vurgulanmaktadır.…”

Section: Uzun öZetunclassified

“…Think-aloud protocols are considered as more effective in understanding raters' cognitive processing during oral assessment scoring because it is sometimes difficult to remember what someone did and why he/she did it (Van Someren, Barnard, & Sandberg, 1994). While employing think-aloud protocols for understanding how raters assign a score can shed light on the rater effects in oral assessment, there is still a limited body of research focusing on thinkaloud in oral interview assessments (e.g., Joe, Harmes, & Hickerson, 2011;Orr, 2002). …”

Section: Introductionmentioning

confidence: 99%

“…Although there are several studies agreeing on the role of the raters' beliefs, perceptions and bias in affecting test results, defining those factors that might influence rater judgment is still in its exploratory stage; and to the knowledge of the researchers, no study has focused on whether raters' prior knowledge of students' proficiency levels may have an impact on their assessment behaviors in oral interviews. Joe (2008), emphasizing the complex procedural and cognitive processes the raters go through while assigning scores in performance assessments, suggests that human scoring involves two important principles: "what raters perceive and how raters think" (p. 4). For this reason, due to the fact that statistical approaches fail in providing a full understanding of the decision making process, recent studies have started to show interest in applying cognitive processing models (e.g., think-alouds) in order to gain better insights into how raters assign scores, and why there are differences among raters' scorings (Brown, 2000).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

Tanriverdi-Köksal¹

2017

HUJE

View full text Add to dashboard Cite

There has been an ongoing debate on the reliability of oral exam scores due to the existence of human raters and the factors that might account for differences in their scorings. This quasi-experimental study investigated the possible effect(s) of the raters' prior knowledge of students' proficiency levels on rater scorings in oral interview assessments. The study was carried out in a pre-and post-test design with 15 EFL instructors who performed as raters in oral assessments at a Turkish state university. In both pre-and post-tests, the raters assigned scores to the same video-recorded oral interview performances of 12 students from three different proficiency levels. While rating the performances, the raters also provided verbal reports about their thought processes. The raters were not informed about the students' proficiency levels in the pre-test, while this information was provided in the posttest. According to the findings, majority of the Total Scores ranked lower or higher in the post-test. The thematic analysis of the raters' video recorded verbal reports revealed that most of the raters referred to the proficiency levels of the students while assigning scores in the post-test. The findings of the study suggest that besides factors such as accent, nationality, and gender of the test-takers and the assessors, raters' prior knowledge of students' proficiency levels could be a variable that needs to be controlled for more reliable test results.Keywords: rater effects, intra-rater reliability, paired oral exams, think-aloud protocols ÖZ: Yaygın olarak kullanılmakta olsa da notlandıran olarak insan faktörünün varlığı ve notlardaki farklılığa neden olan etmenler sebebiyle konuşma sınav notlarının güvenirliği konusunda süregelen bir tartışma vardır. Bu yarı deneysel çalışma, konuşma sınavlarının değerlendirilmesinde, not verenlerin öğrencilerin dil yeterlilik seviyelerini önceden biliyor olmasının verdikleri notları üzerindeki etkilerini araştırmayı amaçlamaktadır. Bu çalışma, Türkiye'deki bir devlet üniversitesinde yabancı dil olarak İngilizce öğreten ve aynı üniversitede konuşma sınavlarında notlandıran olarak görev alan 15 okutman ile ön ve son test olarak iki oturumda yürütülmüştür. Hem ön hem de son testte, notlandıranlar üç farklı seviyeden 12 öğrencinin video kaydına alınmış aynı konuşma sınavı performansları için not vermiştir. Aynı zamanda, performanslar için not verirken, notlandıranlar eş zamanlı olarak ne düşündükleri ile ilgili sözlü bildirimde bulunmuştur. Öğrencilerin dil yeterlilik seviyeleri ile ilgili ön testte herhangi bir bilgi verilmezken, notlandıranlar öğrencilerin seviyeleri konusunda son testte sözlü ve yazılı olarak bilgilendirilmiştir. Sonuçlara göre, önteste kıyasla son testte Toplam Notların büyük çoğunluğunun son testte düştüğü veya yükseldiği saptanmıştır. Tüm notlandıranların video kayıtlı sözlü bildirimleri tematik olarak incelendiğinde, notlandıranların çoğunun son testte not verirken öğrencilerin dil yeterlilik seviyelerine değindikleri gözlemlenmiştir. Çalışmanın ...

show abstract

“…Although holistic rubrics are arguably easier to develop and may reduce grading time, they lack the additional structure of quality descriptors, which not only provide more detailed feedback to students, but also guide raters in their assessment, thus, facilitating consistency in grading 4 . This is particularly important in the development of rubrics for large courses where multiple raters may be involved in assessment.…”

Section: Introductionmentioning

confidence: 99%

Using a Delphi Approach to Develop Rubric Criteria

Lesmond¹,

Dawe²,

Romkey³

et al.

2016 ASEE Annual Conference &Amp; Exposition Proceedings

View full text Add to dashboard Cite

Recent developments in post-secondary institutions have motivated a shift towards outcomesbased education. A major impetus for this agenda has been the growing need to provide concrete evidence of student learning and institutional effectiveness to various stakeholders. Given this trend, it is important that research be undertaken to explore valid approaches to learning outcomes assessment.The research described here involves the development of valid, non-discipline specific, analytic rubrics that assess learning outcomes in five key areas: communication, design, teamwork, problem analysis and investigation. This paper reports on the methodology used to complete the first stage of rubric development; identifying the standards through which student work is evaluated. In particular, a two-stage Delphi study was designed to identify rubric criteria for assessing problem analysis and investigation. The Delphi technique is an iterative research tool used to elicit input from a panel of experts. It typically involves a series of virtual survey rounds in which experts offer their views anonymously and have the opportunity to refine them based on controlled feedback from earlier rounds. Panel members include 11 experts for investigation and 15 experts for problem analysis from faculty and staff. In the first round, participants were asked to propose learning outcome statements or "indicators" that are important for assessing problem analysis or investigation. In the second and final round, these responses were arranged by major outcome areas and sent to participants for their feedback. They were asked to rate how likely they were to use the indicators, and their importance in the curriculum. The focus of this paper is not the results of this study, but the methodological processes involved in designing and administering a Delphi survey to develop tools for learning outcomes assessment. This includes expert selection, survey design, and analysis of expert responses. Special attention is paid to the challenges of conducting a Delphi study.

show abstract

Using verbal reports to explore rater perceptual processes in scoring: a mixed methods application to oral communication assessment

Cited by 19 publications

References 31 publications

An Investigation of Rater Cognition in the Assessment of Projects

An Investigation of Rater Cognition in the Assessment of Projects

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

Using a Delphi Approach to Develop Rubric Criteria

Contact Info

Product

Resources

About