A meta-analysis of interrater and internal consistency reliability of selection interviews.

Abstract

A meta-analysis of 111 interrater reliability coefficients and 49 coefficient alphas from selection interviews was conducted. Moderators of interrater reliability included study design, interviewer training, and 3 dimensions of interview structure (standardization of questions, of response evaluation, and of combining multiple ratings). Interactions showed that standardizing questions had a stronger moderating effect on reliability when coefficients were from separate (rather than panel) interviews, and multiple ratings were useful when combined mechanically (there was no evidence of usefulness when combined subjectively). Average correlations (derived from alphas) between ratings were moderated by standardization of questions and number of ratings made. Upper limits of validity were estimated to be .67 for highly structured interviews and .34 for unstructured interviews.Researchers have long been interested in determining how the reliability and the validity of selection-interview ratings can be improved (e.g., Wagner, 1949). Several recent meta-analyses have examined moderators of interview validity (e.g., Huffcutt & Arthur, 1994;McDaniel, Whetzel, Schmidt, & Maurer, 1994;Weisner & Cronshaw, 1988). These studies have provided useful information, such as the general finding that validity is greater for structured interviews than for unstructured interviews. However, Hakel (1989) proposed another approach to improving interview ratings-studying interview reliability. Hakel suggested a meta-analysis covering interrater coefficients, stabilities over time, and internal consistencies. Note that a reliability coefficient indicating stability over time is an interrater coefficient where the ratings are based on separate interviews (rather than a panel interview) with the same applicant. Therefore, reliabilities can be divided into three groups: two

“I Know What You Want to Know”: The Impact of Interviewees' Ability to Identify Criteria on Interview Performance and Construct-Related Validity

“…Although we cannot rule out this possibility, it seems unlikely because our results are comparable to previous findings concerning interviewer reliability (Conway, Jako, & Goodman, 1995;Taylor & Small, 2002) and interview construct-related validity obtained in field ABILITY TO IDENTIFY CRITERIA samples (Conway & Peneno, 1999;Huffcutt, Weekley, et al, 2001;Schuler & Funke, 1989;Van Iddekinge, Raymark, Eidson, & Attenweiler, 2004).…”

Selling Oneself: Construct and Criterion-Related Validity of Impression Management in Structured Interviews

“…This indicates that the constructs that the interview had intended to measure are of little importance in establishing interviewer impressions of a candidate (Conway, Jako, & Goodman, 1995). At the same time, the current study makes several substantial new contributions.…”

Practical Impact of Predictor Reliability for Personnel Selection Decisions

“…For example, Pulakos, Schmitt, Whitney, and Smith (1996) showed that even for structured interview, which curbs the influence that individual interviewer's intuitions might exert on the procedure given the formal nature of the application and rating process (Campion, Pursell, & Brown, 1988), the differences are likely to be due to chance rather than any real effect, considering the variance in the validity of the individual interviewers' ratings. In addition, Conway, Jako, and Goodman (1995) suggested that because interrater reliability in unstructured interview is so low, its ratings could not account for more than 10% of the variance in the job performance criterion. In support of their findings, Cortina, Goldstein, Payne, Davison, and Gilliand's (2000) meta-analysis showed that unstructured interview rarely contributed to the prediction of job performance above and beyond the variance explained by cognitive ability and conscientiousness (change in R 2 ranging from .01…”