Rater training: Understanding effects of training content, practice ratings, and feedback

Hauenstein, Neil M. A.; McCusker, Maureen E.

doi:10.1111/ijsa.12177

Cited by 8 publications

(6 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We should note that the FOR training seems to be effective, and it should be taken into account in future researches, especially in terms of increasing its effect on the accuracy of data provided by the SMEs, which occurs immediately after the presentation of the purpose and definitions of the dimensions rated by participants (Hauenstein & McCusker, 2017).…”

Section: Practical and Theoretical Implicationsmentioning

confidence: 99%

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

Manea

Iliescu

2021

PRU

View full text Add to dashboard Cite

In this paper we detail the construction and validation process for a new personality-oriented work analysis instrument, in the form of a standardized questionnaire, based on extant research that shows that personality traits are good predictors of job performance. We present the process of item development, frame of reference training, rating scale creation, and the selection of subject matter experts. By administering the instrument to three distinct positions, the interrater reliability coefficients resulted between .80 and .94. We also investigated the instrument’s ability to discriminate between the same rated positions, and the results for this indicator were quite low. Conclusions provide some possible explanations for the lower resulted discriminability. Practical and theoretical implications are discussed as well as other future research for general improvement of data quality.

show abstract

Section: Practical and Theoretical Implicationsmentioning

confidence: 99%

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

Manea

Iliescu

2021

PRU

View full text Add to dashboard Cite

show abstract

“…If the decisions that are made based on test results are vital, rater behaviors should be determined and these behaviors should be reduced to an acceptable level (Ellis et al, 2002). When the literature was examined, it was seen that many rater training designs were suggested and used (Bernardin & Buckley, 1981;Feldman et al, 2012;Haladyna, 1997;Hauenstein, & McCusker, 2017;Stamoulis & Hauenstein, 1993;Weigle, 1998;Zedeck & Cascio, 1982). When the literature is examined, it is seen that there are many rater trainings, but the existence of such a study in the national literature has been the main motivation for conducting this study.…”

Section: Introductionmentioning

confidence: 99%

Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill

Şata

KARAKAYA

2022

International Journal of Assessment Tools in Education

View full text Add to dashboard Cite

In the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individual performance. The study was conducted with a pretest-posttest control group quasi-experimental design. In this research, 45 raters were employed, 23 from the control group and 22 from the experimental group. As data collection tools, a writing task that was developed by IELTS and an analytical rubric that was developed to assess academic writing skills were used. As part of the experimental procedure, rater training was provided and this training was implemented by combining rater error training and frame of reference training. When the findings of the study were examined, it was found that the control and experimental groups were similar to each other before the experiment, however, after the experimental process, the study group made more valid and reliable measurements. As a result, it was investigated that the rater training given had an impact on rater errors such as rater severity, rater leniency, central tendency, and Halo effect. Based on the obtained findings, some suggestions were offered for researchers and future studies.

show abstract

“…Studies have unearthed that rater effects are widespread and their effects can be mitigated by rater training and monitoring efforts (Feldman, Lazzara, Vanderbilt & DiazGranados, 2012;Hauenstein, & McCusker, 2017;Şata, 2019), and have demonstrated that many facet Rasch models (MFRM) can foster the detectability and understandability of the nature of these effects (Engelhard & Myford, 2003;Kim, Park & Kang, 2012;Wolfe, Chiu, & Myford, 2000).…”

Section: Introductionmentioning

confidence: 99%

Examining the Differential Rater Functioning in the Process of Assessing Writing Skills of Middle School 7th Grade Students

Aslanoğlu

Şata

2021

Participatory Educational Research

View full text Add to dashboard Cite

When students present writing tasks that require higher order thinking skills to work, one of the most important problems is scoring these writing tasks objectively. The fact that raters give scores below or above their performance based on several environmental factors affects the consistency of the measurements. Inconsistencies in scoring negatively affect the validity and reliability of student performance and cause the scores obtained to be questioned. In regard to the validity and reliability of these measurements, it is significant to identify the rater behavior and correct the sources of error. This study aims to analyze the differential rater functioning (DRF), which is one of the problematic rater behaviors, in evaluating compositions written by middle school 7th-grade students within the scope of the Turkish course. 86 students attending a public school were participated the study. Students' compositions were rated using an analytical rubric by 8 teachers from different institutions. In this correlational research, the many facet Rasch model was used, and five variables including students, raters’ and, students’ gender, students’ qualification, and evaluation criteria were examined. it was examined whether the raters show DRF on an individual and group basis based on the dual interaction analysis, including the gender of the student x rater and the student's competence x rater. The findings have revealed that DRF at the group level does not interfere with the measurements, while the individual level DRF is involved in the measurements. It was determined that the level of DRF mixing in the measurements of successful students was the lowest. Especially rigid and lenient raters were found to show DRF. In the present study, it was observed that the raters showing DRF was also the most lenient raters, while these raters did not show DRF in terms of the gender of the student.

show abstract

Rater training: Understanding effects of training content, practice ratings, and feedback

Cited by 8 publications

References 49 publications

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill

Examining the Differential Rater Functioning in the Process of Assessing Writing Skills of Middle School 7th Grade Students

Contact Info

Product

Resources

About