Effective Crowd Annotation for Relation Extraction

Liu, Angli; Soderland, Stephen; Bragg, Jonathan; Lin, Christopher H.; Ling, Xiao; Weld, Daniel S.

doi:10.18653/v1/n16-1104

Cited by 64 publications

(75 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the approach generates blocking rules on structured data using random forest, and the method cannot be applied to text data studied in our approach. Crowdsourcing is also applied in relation extraction [24,1]. However, similar to entity matching, most of the works focus on tuplelevel annotation.…”

Section: Related Workmentioning

confidence: 99%

Cost-effective data annotation using game-based crowdsourcing

et al. 2018

View full text Add to dashboard Cite

Large-scale data annotation is indispensable for many applications, such as machine learning and data integration. However, existing annotation solutions either incur expensive cost for large datasets or produce noisy results. This paper introduces a cost-effective annotation approach, and focuses on the labeling rule generation problem that aims to generate high-quality rules to largely reduce the labeling cost while preserving quality. To address the problem, we first generate candidate rules, and then devise a game-based crowdsourcing approach CrowdGame to select high-quality rules by considering coverage and precision. CrowdGame employs two groups of crowd workers: one group answers rule validation tasks (whether a rule is valid) to play a role of rule generator, while the other group answers tuple checking tasks (whether the annotated label of a data tuple is correct) to play a role of rule refuter. We let the two groups play a two-player game: rule generator identifies high-quality rules with large coverage and precision, while rule refuter tries to refute its opponent rule generator by checking some tuples that provide enough evidence to reject rules covering the tuples. This paper studies the challenges in CrowdGame. The first is to balance the trade-off between coverage and precision. We define the loss of a rule by considering the two factors. The second is rule precision estimation. We utilize Bayesian estimation to combine both rule validation and tuple checking tasks. The third is to select crowdsourcing tasks to fulfill the game-based framework for minimizing the loss.We introduce a minimax strategy and develop efficient task selection algorithms. We conduct experiments on entity matching and relation extraction, and the results show that our method outperforms state-of-the-art solutions.

show abstract

Section: Related Workmentioning

confidence: 99%

Cost-effective data annotation using game-based crowdsourcing

et al. 2018

View full text Add to dashboard Cite

show abstract

“…We conducted the RE experiments in this setting, as this allowed us to directly compare with Liu et al (2016). Thus, we used CS A annotated by our proposed method along with the noisy annotated DS to train the extractor.…”

Section: Resultsmentioning

confidence: 99%

“…Similarly to Liu et al (2016), we restricted our attention to 5 relations between person and location 3 . For both DS and CS, we used the publicly available data provided by Liu et al (2016). Ultimately, 221 crowd workers participated to the task with minimum 2 and maximum 400 annotations per crowd worker.…”

Section: Corporamentioning

confidence: 99%

Self-Crowdsourcing Training for Relation Extraction

Abad

Nabi

Moschitti³

2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

One expensive step when defining crowdsourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we introduce a self-training strategy for crowdsourcing. The main idea is to use an automatic classifier, trained on weakly supervised data, to select examples associated with high confidence. These are used by our automatic agent to explain the task to crowd workers with a question answering approach. We compared our relation extraction system trained with data annotated (i) with distant supervision and (ii) by workers instructed with our approach. The analysis shows that our method relatively improves the relation extraction system by about 11% in F1.

show abstract

“…Werling et al (2015) use Bayesian decision theory to minimize human cost and maximize accuracy for named entity recognition. For certain IE tasks, however, human supervision can be very noisy and thus counterproductive, especially from crowds, thus previous work has shown the importance of how to pose tasks for humans in providing labels (Liu et al, 2016), as well as automatically distinguishing simple labeling tasks from expert tasks in crowd-based task assignments (Wang et al, 2017). Our work shares a similar view of human supervision for IE, yet we instead study the impact of the annotation interface on the overall performance.…”

Section: Related Workmentioning

confidence: 98%

Visual Supervision in Bootstrapped Information Extraction

Berger¹,

Nagesh²,

Levine³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We challenge a common assumption in active learning, that a list-based interface populated by informative samples provides for efficient and effective data annotation. We show how a 2D scatterplot populated with diverse and representative samples can yield improved models given the same time budget. We consider this for bootstrapping-based information extraction, in particular named entity classification, where human and machine jointly label data. To enable effective data annotation in a scatterplot, we have developed an embeddingbased bootstrapping model that learns the distributional similarity of entities through the patterns that match them in a large data corpus, while being discriminative with respect to human-labeled and machine-promoted entities. We conducted a user study to assess the effectiveness of these different interfaces, and analyze bootstrapping performance in terms of human labeling accuracy, label quantity, and labeling consensus across multiple users. Our results suggest that supervision acquired from the scatterplot interface, despite being noisier, yields improvements in classification performance compared with the list interface, due to a larger quantity of supervision acquired.

show abstract

Effective Crowd Annotation for Relation Extraction

Cited by 64 publications

References 13 publications

Cost-effective data annotation using game-based crowdsourcing

Cost-effective data annotation using game-based crowdsourcing

Self-Crowdsourcing Training for Relation Extraction

Visual Supervision in Bootstrapped Information Extraction

Contact Info

Product

Resources

About