Restructured frame‐of‐reference training improves rating accuracy

Abstract

Summary The use of heuristic judgments is prevalent in organizations and negatively impacts accurate employee assessments. To minimize the negative impact of heuristic judgments (i.e., anchoring and adjustment), we aim to improve rating accuracy by restructuring frame‐of‐reference (FOR) training. We conducted five studies (N = 1,143) using different samples (three including participants with hiring experience), training environments (onsite and online), and rating contexts (evaluations of sales representatives, teachers, contract negotiation specialists, and retail store managers). Across the five studies, the average improvement in rating accuracy was at least twice as large for restructured FOR (vs. control) training as it was for typical FOR (vs. control) training; the difference in rating accuracy between restructured and typical FOR training was statistically significant. Furthermore, minimizing the anchoring effect rather than increasing opportunities for rating adjustments improved rating accuracy (Study 4). Finally, restructured FOR training achieved higher criterion validity (i.e., a higher strength of the association between ratings regarding a target and the target's objective performance) than did typical FOR training (Studies 3 and 5). We discuss implications for improving the effectiveness of diverse training programs and the accuracy of judgments in organizations.

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

“…After reading and signing an informed consent form, the SMEs participated in a group POWA session that lasted approximately 30 minutes. The SMEs went through the FOR training which began with five hypothetical examples regarding the influence of the personality trait "Sociability" on job performance and the rationale for using the 5 options in the Likert rating scale for each position (Tsai et al, 2019).…”

“…An important first step in implementing our FOR training was the one provided by Aguinis et al (2009), in which they managed to significantly reduce the self-serving bias manifested by the SMEs in a similar POWA process. The other important step in building our FOR training is represented by Tsai's et al (2019) model, respectively a "restructured" type of FOR training. In this type of training, SMEs are first introduced to the "correct" type of rating, and then they are invited to practice the example ratings using the previously learned scheme and only afterwards to rate the items (in our case, the personality traits) of the questionnaire itself.…”

“…We believe that information overload was not an issue, as we have condensed the instrument's contents to a minimum and to its essential. Regarding the heuristics bias, although our FOR training was built in order to diminish the possible effects of anchoring and adjustment heuristics on the model based by Tsai et al, 2019, other two possible types of heuristics might be present, such as representativeness and availability (Morgeson & Campion, 1997). Firstly, we will inform the SMEs in the instructions section of the instrument that they can be unconsciously susceptible to some cognitive errors (Nisbet & Ross, 1980).…”