Toward Better Meta‐analytic Matrices: How Input Values Can Affect Research Conclusions in Human Resource Management Simulations

Abstract

Simulations and analyses based on meta‐analytic matrices are fairly common in human resource management and organizational behavior research, particularly in staffing research. Unfortunately, the meta‐analytic values estimates for validity and group differences (i.e., ρ and δ, respectively) used in such matrices often vary in the extent to which they are affected by artifacts and how accurately the values capture the underlying constructs and the appropriate population. We investigate how such concerns might influence conclusions concerning key issues such as prediction of job performance and adverse impact of selection procedures, as well as noting wider applications of these issues. We also start the process of building a better matrix upon which to base many such simulations and analyses in staffing research. Finally, we offer guidelines to help researchers/practitioners better model human resources processes, and we suggest ways that researchers in a variety of areas can better assemble meta‐analytic matrices.

Practical Impact of Predictor Reliability for Personnel Selection Decisions

“…However, with this advantage of the rank ordering selection method comes the downside of increased potential for adverse impact when cognitive ability testing is involved, as has been demonstrated in a considerable amount of research (e.g., Bobko, Roth, & Potosky, 1999;Cascio et al, 1991;Hunter, Schmidt, & Rauschenberger, 1977;Roth, Switzer, Van Iddekinge, & Oh, 2011;Sackett & Roth, 1991;Sackett, Schmitt, Ellingson, & Kabin, 2001). Organizations may favor a particular alternative (maximum performance vs. diversity) depending on the goals or the purposes of the selection.…”

“…Based on this matrix, Roth et al (2011) then generated predictor and job performance criterion data based on both the uncorrected and corrected matrices and for each matrix conducted a multiple regression analysis by regressing job performance on the predictor variables. Results showed that, as would be expected, using the corrected input matrix greatly increased the multiple correlation coefficient (R = .75 for medium-complexity jobs) compared to the multiple correlation coefficient based on the uncorrected input matrix (R = .48).…”

“…The current simulations are based on a population correlation matrix from Roth et al (2011), containing predictor and criterion variables relevant to personnel selection: cognitive ability, structured interview, conscientiousness, and biodata as predictors, and an overall job performance criterion. (This correlation matrix updates a similar one generated from the meta-analysis by Bobko et al (1999).…”

“…In addition, all other correlations in Roth et al do not take job complexity into account. Roth, Switzer, Van Iddekinge, and Oh (2011) are below the main diagonal. These correlations reflect job incumbent correlations that were corrected for range restriction, with the exception of the biodata-structured interview and biodataconscientiousness correlations, where authors could not find appropriately corrected correlations.…”

“…Recognizing that job complexity has important theoretical implications for understanding criterion-related validity estimates for cognitive ability in employment settings (Hunter 1986), Roth et al (2011) reported two separate validity estimates: one for medium-complexity jobs, and one for low-complexity jobs. They obtained these values by averaging validities by level of job complexity across the studies by Hunter (1986), and Salgado et al (2003a;2003b).…”