Holistic and mechanical combination in psychological assessment: Why algorithms are underutilized and what is needed to increase their use

Abstract

Although mechanical combination results in more valid judgments and decisions than holistic combination, existing publications suggest that mechanical combination is rarely used in practice. Yet, these publications are either descriptions of anecdotal experiences or outdated surveys. Therefore, in several Western countries, we conducted two surveys (total N = 323) and two focus groups to investigate (1) how decision makers in psychological and HR practice combine information, (2) why they do (not) use mechanical combination, and (3) what may be needed to increase its use in practice. Many participants reported mostly using holistic combination, usually in teams. The most common reasons for not using mechanical combination were that algorithms are unavailable in practice and that stakeholders do not accept their use. Furthermore, decision makers do not quantify information, do not believe in research findings on evidence-based decision making, and think that combining holistic and mechanical combination results in the best decisions. The most important reason why mechanical combination is used was to increase predictive validity. To stimulate the use of mechanical combination in practice, our results suggest that decision makers should receive more training on evidence-based decision making, and decision aids supporting the use of mechanical combination should be developed.

Predicting Decision-makers' Algorithm Use

“…For example, organizational psychologists may use standardized tests and interviews to hire new managers, and admissions officers may use grades and personal statements to admit students. In these situations, decision makers typically integrate information based on their expert judgment (Neumann, Niessen, Hurks, & Meijer, 2022;Vrieze & Grove, 2009), but they also sometimes receive algorithmic advice in the form of a performance prediction. An exemplary algorithm's prediction may be an unweighted sum score of a test score and an interview rating.…”

Overcoming algorithm aversion in personnel selection and admissions decisions

“…Decision makers are algorithm averse, partly because they worry about negative stakeholder evaluations (Neumann, Niessen, Hurks, & Meijer, 2022;Nolan et al, 2016). Therefore, we investigated when stakeholders appreciate decision-makers' algorithm use more, and when decision makers worry less about negative stakeholder evaluations when using algorithms in hiring decisions.…”

“…An exemplary algorithm would be to add up a test score and an interview rating, and to hire the candidate(s) with the highest score(s). In practice, decision makers typically combine information holistically (Highhouse, 2008 One important reason why decision makers are algorithm averse is the "threat of technological unemployment" (TOTU, Meehl, 1986 p. 374): Decision makers worry that they provide less value to their organization when using algorithms because the hiring process and outcome is not attributed to their holistic judgment anymore (Burton et al, 2020;Neumann, Niessen, Hurks, & Meijer, 2022;Nolan et al, 2016Nolan et al, , 2020. Specifically, Nolan et al (2016) showed that decision makers are worried that stakeholders (e.g., colleagues, management, and applicants) give them less credit for their hiring decisions (i.e., perceive them as less responsible for and less in control of hiring decisions) when using algorithms rather than their holistic judgment.…”