Rate it again: Using the wisdom of many to improve performance evaluations

Abstract

Much research shows that judgmental estimation could be improved by combining estimates from independent judges as well as within judges. These results have been obtained mostly with judgments about matters of fact, that is, for which there are objective truth criteria. In the present research, we extend these findings to performance evaluations. In a controlled field study, expert judges provided evaluations of a large number of essays written by college applicants taking college entrance tests. The judges were asked to evaluate each essay twice—on two occasions, a week apart. This design allowed us to assess the benefits of two methods of combining evaluations: within rater and across raters. Accuracy gains were obtained with both methods. Although the within‐rater combinations yielded fewer gains than the across‐rater ones, they were still appreciable in comparison with the across rater ones. Our findings extend the class of judgments to which the “wisdom of many” could be applied. These findings are potentially applicable to performance evaluations in social, educational, and employment settings.

A simple cognitive method to improve the prediction of matters of taste by exploiting the within-person wisdom-of-crowd effect

“…To obtain the Estimated opinion, participants had to examine the item from a different perspective than their own opinion; in other words, our method was expected to produce two quasi opinions from participants. This approach for obtaining ‘the wisdom of the inner crowd’ is in line with the methods of previous studies 4 , 20 – 29 . To determine the details of the Estimated opinion—the simulated public opinion—we draw on findings from cognitive and social psychology.…”

“…In daily life, we are often asked questions such as, ‘Do you feel I will like the restaurant you visited?’ or ‘Do you think I will be satisfied with the lecture you attended last year?’ by our friends or peers. How can we accurately predict others’ future satisfaction in such situations 1 – 4 ? An intuitive way is to speculate on how much others would enjoy their experiences.…”

“…the aggregation of their opinions) also increases. Furthermore, recent studies 4 , 26 – 35 demonstrate that an individual can harness the within-person wisdom of the crowd (called ‘the wisdom of the inner crowd’). In these studies, an individual produced multiple responses for a problem (basically regarding the matters of fact , see ‘Discussion’), resulting in the aggregated answer being more accurate than a single one.…”

A Behavioral Design to Reform Italy’s Evaluation Policy

“…The powerful effect of activating perceived professional norms and expectations on experts' behavior-Sah (2017) notes-could encourage evaluators to realize and gradually adhere to professional and ethical standards instead of indulging in selfinterest and bias. Recent experimental research findings suggest that the wisdom of many in judgmental evaluation can combine outsider and insider insights (Barneron et al, 2019) in assessing performance across social, educational, and employment settings. The devil's advocate role in mixed teams also reduces escalation tendencies-that is, people's tendency to stick to conformity pressures.…”

“…The devil's advocate role in mixed teams also reduces escalation tendencies-that is, people's tendency to stick to conformity pressures. A similar instrument is red teaming, where distinct teams have to spot shortcomings through critical thinking to address, for instance, gender biases or other diversity issues both in management and evaluation (Barneron et al, 2019). What these ethically conscious-oriented techniques stress is that an ethical organizational culture can be the only way to eradicate implicit biases when isomorphism pressures are underway (Powell & DiMaggio, 1991) and legal regulation is weak or unenforceable or the awareness of bias is low because of pervasive social norms or organizational routines (March & Olsen, 1989;Sah, 2017).…”