Challenges in Data Crowdsourcing

García-Molina, Héctor; Joglekar, Manas; Marcus, Adam; Parameswaran, Aditya; Verroios, Vasilis

doi:10.1109/tkde.2016.2518669

Cited by 116 publications

(50 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Answer aggregation, thus, shall capture worker characteristics, to assess the likelihood of them providing correct answers and to justify their effects in the aggregated result. The existence of different worker types, as illustrated above, has been verified in various studies [26], [27] as well as in our experimental evaluation. (R2) Support for partial answer validity.…”

Section: Problem Statementsupporting

confidence: 77%

“…Large-Scale Simulation. To evaluate the scalability of our approach in the context of very large crowdsourcing datasets as described in [27], [41], [50], we rely on simulation. To this end, we adapt existing tools [7] for the multi-label setting.…”

Section: Methodsmentioning

confidence: 99%

“…Today's crowdsourcing datasets are very large [27], [41], so that answer aggregation becomes impractical even with a linear inference algorithm. Interactive crowdsourcing applications [42] might require very fast response time.…”

Section: Scalable Model Inference and Predictionmentioning

confidence: 99%

See 2 more Smart Citations

Computing Crowd Consensus with Partial Agreement

Hùng

Viet

Tâm

et al. 2018

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Abstract-Crowdsourcing has been widely established as a means to enable human computation at large-scale, in particular for tasks that require manual labelling of large sets of data items. Answers obtained from heterogeneous crowd workers are aggregated to obtain a robust result. However, existing methods for answer aggregation are designed for discrete tasks, where answers are given as a single label per item. In this paper, we consider partial-agreement tasks that are common in many applications such as image tagging and document annotation, where items are assigned sets of labels. Common approaches for the aggregation of partial-agreement answers either (i) reduce the problem to several instances of an aggregation problem for discrete tasks or (ii) consider each label independently. Going beyond the state-of-the-art, we propose a novel Bayesian nonparametric model to aggregate the partial-agreement answers in a generic way. This model enables us to compute the consensus of partially-sound and partiallycomplete worker answers, while taking into account mutual relationships in labels and different answer sets. We also show how this model is instantiated for incremental learning, incorporating new answers from crowd workers as they arrive. An evaluation of our method using real-world datasets reveals that it consistently outperforms the state-of-the-art in terms of precision, recall, and robustness against faulty workers and data sparsity.

show abstract

Section: Problem Statementsupporting

confidence: 77%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Computing Crowd Consensus with Partial Agreement

Hùng

Viet

Tâm

et al. 2018

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…This is to avoid the possible overestimation of ranking quality caused by the presentation bias aka position bias effect (i.e., users tend to assign greater relevance to higher ranked results) [3,27]. It is challenging to ensure the reliability of relevance judgments obtained from crowdsourcing [17].…”

Section: Discussionmentioning

confidence: 99%

Finding Parallel Passages in Cultural Heritage Archives

Harris

Levene

Zhang

et al. 2018

J. Comput. Cult. Herit.

View full text Add to dashboard Cite

It is of great interest to researchers and scholars in many disciplines (particularly those working on cultural heritage projects) to study parallel passages (i.e., identical or similar pieces of text describing the same thing) in digital text archives. Although there exist a few software tools for this purpose, they are restricted to a specific domain (e.g., the Bible) or a specific language (e.g., Hebrew). In this paper, we present in detail how we build a digital infrastructure that can facilitate the search and discovery of parallel passages for any domain in any language. It is at the core of our Samtla (Search And Mining Tools with Linguistic Analysis) system designed in collaboration with historians and linguists. The system has already been used to support research on five large text corpora that span a number of different domains and languages. The key to such a domain-independent and language-independent digital infrastructure is a novel combination of a character-based n-gram language model, space-optimised suffix tree, generalised edit distance. A comprehensive evaluation through crowd-sourcing shows that the effectiveness of our system's search functionality is on par with the human-level performance.

show abstract

“…Accommodating them in a generic set of learning objectives is an essential part of providing the customized learning paths. We attempted to convert each learning unit into a set of data science competences using data crowdsourcing [3], [4]. Crowdsourcing refers to solving large problems by involving human workers that solve component of subproblems or tasks [3], while data crowdsourcing particularly focuses on enriching and expanding the initial data set.…”

Section: Introductionmentioning

confidence: 99%

Tailored Data Science Education Using Gamification

Zicari

Tolle

Manieri

2016

2016 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)

View full text Add to dashboard Cite

Abstract-Interest to become a data scientist or related professions in data science domain is rapidly growing. To meet such a demand, we propose a novel educational service that aims to provide tailored learning paths for data science. Our target user is one who aims to be an expert in data science. Our approach is to analyze the background of the practitioner and match the learning units. A critical feature is that we use gamification to reinforce the practitioner engagement. We believe that our work provides a practical guideline for those who want to learn data science.

show abstract

Challenges in Data Crowdsourcing

Cited by 116 publications

References 42 publications

Computing Crowd Consensus with Partial Agreement

Computing Crowd Consensus with Partial Agreement

Finding Parallel Passages in Cultural Heritage Archives

Tailored Data Science Education Using Gamification

Contact Info

Product

Resources

About