Crowd Research: Open and Scalable University Laboratories

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

“…Although this approach has received increasing attention by enabling thousands of people spatially distributed "to coordinate joint progress on an openended research effort" [4], some researchers are still reluctant to adopt crowdsourcing technology due to the highly uncertain and iterative nature of scientific discovery [3,10]. This breadth raises a number of new challenges, including instruction ambiguity and worker honesty [15], motivation for participation in non-profit communities [16], and limited expertise and attention to cope with high-dimensional and ill-structured data [17].…”

“…This evidence further highlight several potential opportunities for CSCW research to better understand how and why crowds are communicating in these subcommunities as well as the potential impact of these interactions (e.g., crowd bias). In crowd science, open projects may need a higher degree of collaboration [11], and the nature and roles of crowd members in research settings can vary from a core scientist/principal researcher (who advices the research project) to an academic expert from an external institution, local case actors (e.g., residents affected by the issue under consideration), individual scientists performing preliminary research work, and volunteers or members of the crowd who have different capabilities and use computing devices to interact, coordinate, and execute tasks [4].…”

“…Crowd science has been deployed using direct engagement and (explicit) peer feedback to train crowd participants [4]. The level of activeness in the crowdsourcing system and the relationship between researchers (as requesters) and crowd workers (or volunteers) are crucial for a research project's success [52].…”

“…For humans, the massive volume of new research findings makes it impossible to grasp due to the human cognitive and memory limits. On the other hand, algorithms still cannot process efficiently the scientific unstructured data [4]. To bridge this gap, researchers have slightly explored the applicability of hybrid crowd-machine methods and techniques, mostly in small-scale institutional settings.…”

A Simple Computational Theory of General Collective Intelligence

“…Researchers in crowdsourcing and human computation have repeatedly demonstrated the ability of ad‐hoc human groups to accomplish tasks they have no prior experience executing. Successful examples range from scaffolded collaboration in crowdsourcing applications such as robotic control (Lasecki, Murray, White, Miller, & Bigham, ), scientific research (Vaish et al., ), or animation (Lasecki et al., ) to fairly open‐ended collaboration in stylized games such as multiagent tracking (Krafft, Hawkins, Pentland, Goodman, & Tenenbaum, ) and real applications like on‐the‐fly disaster response crisis mapping (Mao, Mason, Suri, & Watts, ).…”