CrowdForge

Kittur, Aniket; Smus, Boris; Khamkar, Susheel; Kraut, Robert E.

doi:10.1145/1979742.1979902

Cited by 102 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Work on user interfaces which require some level of accuracy often tries to minimize unnecessary cognitive load [19,40,51]. Similarly, work in crowdsourcing recommends to chunk difficult tasks into smaller units of work [28]. Some work has shown that crowdsource platforms have great potential for rapidly collecting measurements in user studies [27].…”

Section: Related Workmentioning

confidence: 99%

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg,

Dhingra,

Yang

2023

Preprint

View full text Add to dashboard Cite

Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers' F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not -achieving mean F1 of 0.70. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg,

Dhingra,

Yang

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The task is broken into multiple subtasks that are concurrently solved and verified by other workers, and eventually merged into a cumulative output. However, although the framework relies on a powerful paradigm of parallel work execution, it assumes that complex work can be decomposed into lots of merely dependent micro tasks -an assumption that is often violated [12]. Other notable examples of online collaboration in crowdsourcing are CrowdWeaver -supporting with visual interface for real-time managing both human and machine crowdtasks within an integrated workflow [13],…”

Section: Online Collaborationmentioning

confidence: 99%

“…were not able to confirm this statistically. It is unclear whether this is due to a small sample of respondents (12). Future work might explore this by increasing the sample size and with recording additional data indicating the complexity of tasks.…”

Section: Limitations and Future Workmentioning

confidence: 99%

Towards Collaborative Data Analysis with Diverse Crowds – A Design Science Approach

Feldman

Anastasiu

Bernstein

2018

Designing for a Digital and Globalized World

View full text Add to dashboard Cite

The last years have witnessed an increasing shortage of data experts capable of analyzing the omnipresent data and producing meaningful insights. Furthermore, some data scientists mention data preprocessing to take up to 80% of the whole project time. This paper proposes a method for collaborative data analysis that involves a crowd without data analysis expertise. Orchestrated by an expert, the team of novices conducts data analysis through iterative refinement of results up to its successful completion. To evaluate the proposed method, we implemented a tool that supports collaborative data analysis for teams with mixed level of expertise. Our evaluation demonstrates that with proper guidance data analysis tasks, especially preprocessing, can be distributed and successfully accomplished by non-experts. Using the design science approach, iterative development also revealed some important features for the collaboration tool, such as support for dynamic development, code deliberation, and project journal. As such we pave the way for building tools that can leverage the crowd to address the shortage of data analysts.

show abstract

“…Skopik et al investigate interaction patterns for trust-based routing of tasks among interconnected experts [55]. Kittur et al outline how the Map-Reduce architectural style enables crowdsourcing of complex workflows [56]. Dustdar and Gaedke propose context-dependent selection of collaboration structures for task delegation [57].…”

Section: Related Workmentioning

confidence: 99%

Analyzing runtime adaptability of collaboration patterns

Dorn

Taylor

2012

2012 International Conference on Collaboration Technologies and Systems (CTS)

View full text Add to dashboard Cite

The recent two decades have witnessed the emergence of large-scale, interaction-intensive systems. A system's provided user-centric communication and coordination mechanisms have a significant impact on its runtime management. Beyond a certain size, manual monitoring and management are no longer feasible. Hence, it is highly important for a system designer to becoming aware of the most suitable interaction mechanisms and their implications on system adaptability. Specifically, a system designer requires knowledge on what adaptation primitives are available, whether these are system-driven or user-driven, how long they will take, what impact do they have on collaboration state, and under what conditions they can be enacted. These aspects vary considerably across collaboration patterns. In this paper, we investigate a collaboration structure's adaptability based on behavior, asynchrony, state, and execution context. We subsequently discuss seven distinctively different collaboration patterns in terms of those aspects. Based on a motivating scenario, we ultimately demonstrate how these patterns and insights into their inherent adaptability may guide design decision impact and trade-off analysis.

show abstract

CrowdForge

Cited by 102 publications

References 3 publications

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Towards Collaborative Data Analysis with Diverse Crowds – A Design Science Approach

Analyzing runtime adaptability of collaboration patterns

Contact Info

Product

Resources

About