Task bias and the accuracy of judgment: Setting a baseline for expected group performance

Abstract

When a judgment task evokes unbiased estimates (i.e. the errors in individual judgments are distributed randomly around the true value), mathematical aggregation of individual estimates, even by a simple arithmetic mean, often will outperform all group members. However, when a task evokes biased estimates, mathematical aggregation does not perform so well. In this study, simulated data were accumulated to specify the expected' accuracy of mathematical aggregation relative to the accuracy of observed judgment of individual group members under varying conditions of task bias. Three types of judgment tasks were employed: (1) single‐estimate, holistic tasks, (2) multiple‐estimate, ranking tasks, and (3) multi‐cue, decomposed tasks. Findings indicated across all task types that a large percentage of judgment‐making group estimates formed strictly by computing the arithmetic mean of individual estimates performed better than their most capable members when a judgment task evoked little or no bias, a result particularly pronounced for ranking tasks. When the task was more greatly bias‐evoking, a large percentage of parallel groups performed more poorly than average (or median) members, again a pattern more starkly evident for ranking tasks. These results suggest that the extent to which a judgment task evokes bias in a population of prospective group members is an important explanatory variable deserving much greater attention in the study of group performance. For example, an assertion about the efficacy of a particular group intervention based on a reliable demonstration of group performance as accurate as the most capable members may be unfounded when a task evokes no bias, since the baseline standard under such conditions should be much higher. By selecting tasks and populations that jointly produced highly biased estimates, researchers can lower the performance floor enough to detect (with reasonably small samples of groups) experimental effects should they occur.

Proper analysis of the accuracy of group judgments.

“…In terms of a group judgment policy, that adjustment is equivalent to decreasing the weights on all of the judgment cues. Reagan-Cirincione and Rohrbaugh (1992;Einhorn, Hogarth, & Klempner, 1977) have argued that systematic judgment error is an important influence on the difficulty of a group judgment task. If the accuracy of the most accurate member's judgment is the baseline accuracy, then the group is more likely to ' 'beat'' that baseline when individual judgments are unbiased.…”

The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance

“…Faithful transfer of the judgment model would lead to high judgment consistency, but not necessarily to accurate results. Accurate results would only be achieved if the model were applied appropriately, in particular, without bias [4,33]. For example, a real estate appraiser with London expertise should predict prices consistently and accurately for London, and consistently, but not necessarily accurately for Liverpool or Manchester (in case of a possible "London bias").…”

The Effect of an Online Self-Assessment Tool on Nonprofit Board Performance

“…The more the respondent agrees with an item, the more the item is seen as an issue facing the board. A diagnostic instrument that measures the extent to which performance issues are perceived to exist at Time 1 is important in that it establishes a baseline measure of governance behavior against which the efficacy of the online tool and reporting system on subsequent board performance improvement decisions can be measured at Time 2 (Reagan-Cirincione & Rohrbaugh, 1992).…”