Strategies for crowdsourcing social data analysis

Abstract

Web-based social data analysis tools that rely on public discussion to produce hypotheses or explanations of patterns and trends in data rarely yield high-quality results in practice. Crowdsourcing offers an alternative approach in which an analyst pays workers to generate such explanations. Yet, asking workers with varying skills, backgrounds and motivations to simply "Explain why a chart is interesting" can result in irrelevant, unclear or speculative explanations of variable quality. To address these problems, we contribute seven strategies for improving the quality and diversity of worker-generated explanations. Our experiments show that using (S1) feature-oriented prompts, providing (S2) good examples, and including (S3) reference gathering, (S4) chart reading, and (S5) annotation subtasks increases the quality of responses by 28% for US workers and 196% for non-US workers. Feature-oriented prompts improve explanation quality by 69% to 236% depending on the prompt. We also show that (S6) pre-annotating charts can focus workers' attention on relevant details, and demonstrate that (S7) generating explanations iteratively increases explanation diversity without increasing worker attrition. We used our techniques to generate 910 explanations for 16 datasets, and found that 63% were of high quality. These results demonstrate that paid crowd workers can reliably generate diverse, high-quality explanations that support the analysis of specific datasets.

Structuring, Aggregating, and Evaluating Crowdsourced Design Critique

“…This general approach is inspired by similar efforts in the HCI community to produce other types of design principles (e.g. [20,28,29]). …”

Cloud-Sourcing: Using an Online Labor Force to Detect Clouds and Cloud Shadows in Landsat Images

“…A poorer performance of non-U.S. workers was also found by [27] in the context of annotating charts and other visualizations. They suggest language barriers as a possible explanation.…”

Extracting references between text and charts via crowdsourcing

“…We consider prior work from both the visualization and natural language processing (NLP) communities in the design of our pipeline. In the context of visualization, researchers have used crowdsourcing to gather data for graphical perception experiments [12,14,20], and to perform data analysis on charts [33]. In the NLP community, Snow et al [31] have demonstrated the viability of crowdsourcing to cheaply generate large sets of labeled training examples for a variety of text analysis tasks.…”

“…There has been much recent work on designing crowdsourcing pipelines for complex tasks including taxonomy creation [8], explaining outliers and trends in data analysis [33], and generating answers to uncommon queries [3]. Others have designed toolkits such as TurKit [23], CrowdForge [18], Turkomatic [21], and Jabberwocky [1], to help developers implement complex crowdsourced pipelines.…”