Let’s Talk it Out: The Effects of Calibration Meetings on Performance Ratings

Speer, Andrew B.; Tenbrink, Andrew P.; Schwendeman, Michael G.

doi:10.1080/08959285.2019.1609477

Cited by 8 publications

(5 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, research has investigated cognitive factors that are likely to positively affect the psychometric properties of performance ratings, including using rater diaries (DeNisi & Peters, 1996) and more sophisticated rating scale formats (Hoffman et al, 2012). Other interventions include the use of calibration meetings (Speer et al, 2019), increasing rater accountability (Mero & Motowidlo, 1995; Roch, 2006; Tenbrink & Speer, 2022), using raters with higher cognitive ability (Speer, 2020), and conducting rater training (Roch et al, 2012; Woehr & Huffcutt, 1994). Although the individual impact of any one design feature will likely only produce marginal gains, when all these features are incorporated simultaneously, performance ratings should exhibit more favorable psychometric properties.…”

Section: Discussionmentioning

confidence: 99%

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Speer,

Delacruz,

Wegmeyer

et al. 2024

Journal of Applied Psychology

View full text Add to dashboard Cite

Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Speer,

Delacruz,

Wegmeyer

et al. 2024

Journal of Applied Psychology

View full text Add to dashboard Cite

show abstract

“… 10. Although there is ample evidence that managers dislike performance appraisals (e.g., Pulakos et al, 2015), PAs are also likely to elicit a wide range of motivations that result in purposeful managerial behavior. Mechanisms such as accountability (e.g., Harari & Rudolph, 2017), calibration meetings (Speer, Tenbrink, & Schwendeman, 2019), and the naturally complex combination of social, self-serving, and organizationally driven motives (e.g., Longenecker, Sims, & Gioia, 1987; Spence & Keeping, 2013) all make the task of giving operational appraisals a purposeful one. …”

mentioning

confidence: 99%

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Speer

2020

Organizational Research Methods

Self Cite

View full text Add to dashboard Cite

Performance appraisal narratives are qualitative descriptions of employee job performance. This data source has seen increased research attention due to the ability to efficiently derive insights using natural language processing (NLP). The current study details the development of NLP scoring for performance dimensions from narrative text and then investigates validity and generalizability evidence for those scores. Specifically, narrative valence scores were created to measure a priori performance dimensions. These scores were derived using bag of words and word embedding features and then modeled using modern prediction algorithms. Construct validity evidence was investigated across three samples, revealing that the scores converged with independent human ratings of the text, aligned numerical performance ratings made during the appraisal, and demonstrated some degree of discriminant validity. However, construct validity evidence differed based on which NLP algorithm was used to derive scores. In addition, valence scores generalized to both downward and upward rating contexts. Finally, the performance valence algorithms generalized better in contexts where the same qualitative survey design was used compared with contexts where different instructions were given to elicit narrative text.

show abstract

“…Towards this end, recent PA scholarship suggests paying more attention to rater disposition and PA context (e.g. Bernardin et al, 2016;Cho et al, 2022;Ellington and Wilson, 2017;Speer et al, 2019;Schleicher et al, 2018;Tenbrink and Speer, 2022). Thus, in the present study, we responded to the above call and found the following:…”

Section: Discussionmentioning

confidence: 50%

“…Overall, by virtue of its design, our study contributes to the existing PA literature that relates rater disposition and PA context with rating distortions (Bernardin et al, 2016;Cho et al, 2022;Ellington and Wilson, 2017;Speer et al, 2019). Raters watched two video tapes that filmed both good and poor performance levels.…”

Section: Discussionmentioning

confidence: 95%

Obliterating rating distortions: role of rater disposition and rating context

Ikramullah

Khalil

Iqbal

et al. 2022

View full text Add to dashboard Cite

PurposeRecent performance appraisal (PA) literature suggests that alongside cognitive biases, rating distortions may stem from rater disposition and PA context. The study investigated the role of social value orientation (rater disposition), PA purposes and rater accountability (PA context) toward rating distortions at both performance levels, i.e. good and poor.Design/methodology/approachThe authors designed an experimental study and elicited data from N = 110 undergraduate students about two video-taped performances of good and poor performers. In these videos, two managers conducted assessment interviews of two different employees for the job of a sales representative at an information technology organization. To ensure the validity of performance ratings, the authors invited 10 senior managers to provide benchmark ratings of the video-taped performances. While being placed in two separate groups, the study participants gave performance ratings on both the video-taped performances. The authors used repeated-measures analysis to analyze data.FindingsThe results revealed that rating distortions took place not because of rater social value orientation, but the PA context. Different rating distortion patterns emerged for different levels of ratees' performance.Originality/valueThis study’s findings furnish new insights for assessing rating distortions for poor as well as good performers. Moreover, the results support previous findings that for good performers, accountable raters are tempted toward accurate ratings and refrained from deflation. Similarly, for poor performers, accountable raters do not inflate ratings. The findings will open research avenues to examine the role of PA purposes in rating distortions for different performance levels.

show abstract

Let’s Talk it Out: The Effects of Calibration Meetings on Performance Ratings

Cited by 8 publications

References 57 publications

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Obliterating rating distortions: role of rater disposition and rating context

Contact Info

Product

Resources

About