Wisdom of Microcrowds in Evaluating Solutions to Esoteric Engineering Problems

Abstract

A multitude of studies in economics, psychology, political and social sciences have demonstrated the wisdom of crowds (WoC) phenomenon, where the collective estimate of a group can be more accurate than estimates of individuals. While WoC is observable in such domains where the participating individuals have experience or familiarity with the question at hand, it remains unclear how effective WoC is for domains that traditionally require deep expertise or sophisticated computational models to estimate objective answers. This work explores how effective WoC is for engineering design problems that are esoteric in nature, that is, problems (1) whose solutions traditionally require expertise and specialized knowledge, (2) where access to experts can be costly or infeasible, and (3) in which previous WoC studies with the general population have been shown to be highly ineffective. The main hypothesis in this work is that in the absence of experts, WoC can be observed in groups that consist of practitioners who are defined to have a base familiarity with the problems in question but not necessarily domain experts. As a way to emulate commonly encountered engineering problem-solving scenarios, this work studies WoC with practitioners that form microcrowds consisting of 5–15 individuals, thereby giving rise to the term the wisdom of microcrowds (WoMC). Our studies on design evaluations show that WoMC produces results whose mean is in the 80th percentile or better across varying crowd sizes, even for problems that are highly nonintuitive in nature.

Exploring the Application of Network Analytics in Characterizing a Conceptual Design Space

“…Crowdsourcing minimizes the workload, especially when rating a large corpus of designs. However, localizing individuals with sufficient levels of expertise remains a fundamental issue, particularly for complicated or nuanced t asks relevant to engineering design (Burnap et al, 2015;Gercer Ulu et al, 2018). Similarly, concerns with trust and quality control exist, as well as the manner in which research investigators frame the task and incentivize their crowd workers.…”