Investigating the Construct Validity of Performance Comments: Creation of the Great Eight Narrative Dictionary

Speer, Andrew B.; Schwendeman, Michael G.; Reich, Caitlynn C.; Tenbrink, Andrew P.; Siver, Sydney R.

doi:10.1007/s10869-018-9599-9

Cited by 10 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although these studies highlight new ways to understand this rich PA data source, there are challenges. Specifically, although it is desirable to depict employee performance according to performance dimensions, Speer et al's (2019) Great 8 theme scores only reflect the degree to which a given performance dimension is discussed; they do not indicate whether the employee is discussed favorably (i.e., whether someone performed well on a given dimension). These theme scores were accompanied by the creation of valence scores intended to reflect employee standing on performance dimensions.…”

mentioning

confidence: 99%

“…This has occurred not only in the PA context but also across the organizational sciences more generally (e.g., Banks et al, 2018; Campion et al, 2016; Kobayashi et al, 2018; Pandey & Pandey, 2019). Within the PA context specifically, recent work by Speer and colleagues (Speer, 2018; Speer, Schwendeman, Reich, Tenbrink, & Siver, 2019) demonstrated that performance-related variance can be captured from performance narratives using NLP, and that variance predicts future behavioral outcomes. Furthermore, although most NLP methods are highly empirical in approach, deductive procedures have been used to score a priori performance dimensions from performance narratives.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Speer

2020

Organizational Research Methods

Self Cite

View full text Add to dashboard Cite

Performance appraisal narratives are qualitative descriptions of employee job performance. This data source has seen increased research attention due to the ability to efficiently derive insights using natural language processing (NLP). The current study details the development of NLP scoring for performance dimensions from narrative text and then investigates validity and generalizability evidence for those scores. Specifically, narrative valence scores were created to measure a priori performance dimensions. These scores were derived using bag of words and word embedding features and then modeled using modern prediction algorithms. Construct validity evidence was investigated across three samples, revealing that the scores converged with independent human ratings of the text, aligned numerical performance ratings made during the appraisal, and demonstrated some degree of discriminant validity. However, construct validity evidence differed based on which NLP algorithm was used to derive scores. In addition, valence scores generalized to both downward and upward rating contexts. Finally, the performance valence algorithms generalized better in contexts where the same qualitative survey design was used compared with contexts where different instructions were given to elicit narrative text.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Speer

2020

Organizational Research Methods

Self Cite

View full text Add to dashboard Cite

show abstract

“…Apart from the theoretical contributions mentioned above, our results extend the literature on selection interviews by using a new type of data – interviewers’ narrative comments – as well as employees’ actual performance data within the company, to illustrate the direct effect of asking job-related questions on interview validity. Moreover, the use of text analysis on interview narrative comments is a method that has not been used in the selection interview literature, even though it is gaining more attention in other areas of human resources management research such as performance appraisal (e.g., Brutus, 2010 ; Speer, 2018 ; Speer et al, 2018 ), applicants’ justice perceptions ( Walker et al, 2015 ), and training of interviewers ( Shantz and Latham, 2012 ). Our study extends selection interview literature by introducing a new method proven in another field of study.…”

Section: Discussionmentioning

confidence: 99%

“…They serve as a tool for interviewers to give a review of the candidate’s response to the questions the interviewer asked during the interview. Since interviewer narrative comments are a kind of performance appraisal that assesses people’s behaviors, they can be scored into different performance dimensions ( Speer et al, 2018 ). Interviewers in our research are required to write notes after each interview by their human resources system.…”

Section: Conceptual Background and Hypotheses Developmentmentioning

confidence: 99%

Predictive Validity of Interviewer Post-interview Notes on Candidates’ Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company

et al. 2021

View full text Add to dashboard Cite

Despite the popularity of the employment interview in the employee selection literature and organizational talent selection process, few have examined the comments interviewers give after each interview. This study investigated the predictability of the match between interviewer post-interview notes and radar charts from job analysis on the candidate’s later career performance using text mining techniques and data from one of the largest internet-based technology companies in China. A large sample of 7,650 interview candidates who passed the interviews and joined the company was obtained to show that the number of job-related capabilities interviewers mentioned in their notes was positively related to candidate’s job performance, the number of promotions, and negatively related to turnover. Moreover, the dimensions of the radar chart from job analysis covered in the interview moderated the predictability of interview post-interview notes. Our results indicated that a smaller number of radar chart dimensions by which interviewers assessed the candidates in the interview positively moderated candidates’ promotion for product development jobs, and negatively moderated turnover for technical jobs. The implications of these results for structured interview research in both theory and practice are discussed.

show abstract

“…Our primary goal was to develop, validate, and freely provide NLP algorithms capable of assessing work attitudes and perceptions. These algorithms were designed to measure the topics a particular text discusses (i.e., theme scores, Speer, 2020; Speer et al, 2019)—similar to Linguistic Inquiry Word Count (LIWC, Tausczik & Pennebaker, 2010) but developed explicitly for work-related attitudes—as well as the degree to which a topic is discussed favorably (i.e., valence scores, Speer, 2020). By relying on nonwork-related NLP dictionaries such as LIWC, researchers are unable to automatically assess which work attitudes and perceptions are discussed in text.…”

Section: Targeted Work Attitudes and Perceptionsmentioning

confidence: 99%

Turning words into numbers: Assessing work attitudes using natural language processing.

Perrotta¹,

Tenbrink²,

Speer³

et al. 2023

Journal of Applied Psychology

Self Cite

View full text Add to dashboard Cite

Researchers and practitioners are often interested in assessing employee attitudes and work perceptions. Although such perceptions are typically measured using Likert surveys or some other closed-end numerical rating format, many organizations also have access to large amounts of qualitative employee data. For example, open-ended comments from employee surveys allow workers to provide rich and contextualized perspectives about work. Unfortunately, there are practical challenges when trying to understand employee perceptions from qualitative data. Given this, the present study investigated whether natural language processing (NLP) algorithms could be developed to automatically score employee comments according to important work attitudes and perceptions. Using a large sample of employees, algorithms were developed to translate text into scores that reflect what comments were about (theme scores) and how positively targeted constructs were described (valence scores) for 28 work constructs. The resulting algorithms and scores are labeled the Text-Based Attitude and Perception Scoring (TAPS) dictionaries, which are made publicly available and were built using a mix of count-based scoring and transformer neural networks. The psychometric properties of the TAPS scores were then investigated. Results showed that theme scores differentiated responses based on their likelihood to discuss specific constructs. Additionally, valence scores exhibited strong evidence of reliability and validity, particularly, when analyzed on text responses that were more relevant to the construct of interest. This suggests that researchers and practitioners should explicitly design text prompts to elicit construct-related information if they wish to accurately assess work attitudes and perceptions via NLP.

show abstract

Investigating the Construct Validity of Performance Comments: Creation of the Great Eight Narrative Dictionary

Cited by 10 publications

References 31 publications

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Predictive Validity of Interviewer Post-interview Notes on Candidates’ Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company

Turning words into numbers: Assessing work attitudes using natural language processing.

Contact Info

Product

Resources

About