Quantifying with words: An investigation of the validity of narrative‐derived performance scores

Abstract

Performance appraisal research has focused almost entirely on traditional numerical ratings despite narrative text comments regularly being collected within appraisals. This study investigated the theory and utility of leveraging narrative comments to better understand employee performance. Narrative sentiment scores were derived using text mining on a large sample of narrative comments. These scores were then applied to an independent set of 2 years of performance data. It was assumed that narrative comments would reflect true performance variance that overlaps with traditional ratings, but also that they would capture incremental variance due to increases in total information and a reduction in rater‐motivated biases in contexts in which narrative data were not explicitly linked to administrative outcomes. The derived narrative scores were reliable across years, converged with traditional numerical ratings and explained incremental variance in future performance outcomes (performance ratings, involuntary turnover, promotions, and pay increases). Collectively, this study highlights how narratives can enhance performance measurement and demonstrates how these data can be economically scored in applied settings.

Statesmen or cheerleaders? Using topic modeling to examine gendered messages in narrative developmental feedback for leaders

“…Feedback leads to increased self-awareness and a more accurate perception of one's performance and ability compared to others (Bollich, Johannet & Vazire, 2011;Kluger & DeNisi, 1996;Maki, 1998), and in the case of leaders, the availability of feedback enhances on-the-job learning (Halpern, 2004), and helps individuals make sense of, and learn from, developmental challenges (DeRue & Wellman, 2009). Although evaluative feedback is used widely in organizational settings (Bracken, Rose & Church, 2016;Kluger & DeNisi, 1996), most research has focused on performance feedback that uses quantitative ratings to assess and compare an individual's past performance against the performance of others or a norm (Speer, 2018). Much less research has investigated developmental feedback, despite its prevalence and popularity in organizations and particularly in leadership development programs (Brutus, 2010).…”

Justice perceptions of artificial intelligence in selection

“…AI techniques have the potential to streamline various HR activities. For example, text mining can be used for screening accomplishment records (Campion, Campion, Campion, & Reider, 2016) and for examining narrative performance comments (Speer, 2018). AI can also be used to screen job application form data to quantify work experience (Sajjadiani, Sojourner, Kammeyer‐Mueller, & Mykerezi, 2019).…”

Predictive Validity of Interviewer Post-interview Notes on Candidates’ Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company

“…Apart from the theoretical contributions mentioned above, our results extend the literature on selection interviews by using a new type of data – interviewers’ narrative comments – as well as employees’ actual performance data within the company, to illustrate the direct effect of asking job-related questions on interview validity. Moreover, the use of text analysis on interview narrative comments is a method that has not been used in the selection interview literature, even though it is gaining more attention in other areas of human resources management research such as performance appraisal (e.g., Brutus, 2010 ; Speer, 2018 ; Speer et al, 2018 ), applicants’ justice perceptions ( Walker et al, 2015 ), and training of interviewers ( Shantz and Latham, 2012 ). Our study extends selection interview literature by introducing a new method proven in another field of study.…”