Adaptive Rule Discovery for Labeling Text Data

Galhotra, Sainyam; Golshan, Behzad; Tan, Wang-Chiew

doi:10.1145/3448016.3457334

Cited by 12 publications

(11 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Uncertainty Sampling (US) [15]: US is a classic and competitive method within active learning paradigm [29]. • IWS-LSE [6]: IWS-LSE is a representative method under the interactive weak supervision paradigm considered in [6,12], where the user is iteratively queried to provide feedback on whether a suggested labeling heuristic is useful or not.…”

Section: Nemo End-to-end System Performancementioning

confidence: 99%

“…• Under-formalized LF Development Workflow: The lack of formalism on the LF development process has obscured systematic study to optimize the workflow, making it less organized and more challenging for practitioners to design LFs for DP applications [6,12,33,35]. • Inefficient Development Data Selection: Current LF development workflow selects development data with the most straightforward approach, uniform random sampling, which unfortunately can be time-consuming as it oftentimes requires users to inspect a considerable amount of data samples to create an informative set of LFs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Hsieh¹,

Zhang²,

Ratner³

2022

Preprint

View full text Add to dashboard Cite

Weak Supervision (WS) techniques allow users to efficiently create large training datasets by programmatically labeling data with heuristic sources of supervision. While the success of WS relies heavily on the provided labeling heuristics, the process of how these heuristics are created in practice has remained under-explored. In this work, we formalize the development process of labeling heuristics as an interactive procedure, built around the existing workflow where users draw ideas from a selected set of development data for designing the heuristic sources. With the formalism, we study two core problems of how to strategically select the development data to guide users in efficiently creating informative heuristics, and how to exploit the information within the development process to contextualize and better learn from the resultant heuristics (Figure 1). Building upon two novel methodologies that effectively tackle the respective problems considered, we present Nemo, an end-to-end interactive system that improves the overall productivity of WS learning pipeline by an average 20% (and up to 47% in one task) compared to the prevailing WS approach.

show abstract

Section: Nemo End-to-end System Performancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Hsieh¹,

Zhang²,

Ratner³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Snuba [91] generates heuristics based on a small set of labeled datasets. [7] and [25] interactively generate labeling functions based on user feedback. TALLOR [46] and GLaRA [106] automatically augment an initial set of labeling functions with new ones.…”

Section: Related Workmentioning

confidence: 99%

“…(2) Active generation and repurposing of supervision sources. To further reduce human annotation efforts, very recently, researchers turn to active generation [91,46,106,7,25] and repurposing [27] of supervision sources. In the future, we plan to incorporate these new tasks and methods into WRENCH to extend its scope.…”

Section: A3 Hosting and Maintenance Planmentioning

confidence: 99%

WRENCH: A Comprehensive Benchmark for Weak Supervision

Zhang,

Yu,

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple potentially noisy supervision sources. However, proper measurement and analysis of these approaches remain a challenge. First, datasets used in existing works are often private and/or custom, limiting standardization. Second, WS datasets with the same name and base data often vary in terms of the labels and weak supervision sources used, a significant "hidden" source of evaluation variance. Finally, WS studies often diverge in terms of the evaluation protocol and ablations used. To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches. It consists of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods. We use WRENCH to conduct extensive comparisons over more than 100 method variants to demonstrate its efficacy as a benchmark platform. The code is available at https://github.com/JieyuZ2/wrench.Preprint. Under review.

show abstract

“…Anchor learning [27] can also suggest new selfsupervision for human review. Darwin [42] incorporates active learning for verifying proposed rules, but it doesn't conduct structure learning, and like Snuba and other data programming methods, it only models individual instances.…”

Section: Task-specific Self-supervisionmentioning

confidence: 99%

Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning

Poon,

Wang,

Lang

2021

Preprint

View full text Add to dashboard Cite

Deep learning has proven effective for various application tasks, but its applicability is limited by the reliance on annotated examples. Self-supervised learning has emerged as a promising direction to alleviate the supervision bottleneck, but existing work focuses on leveraging co-occurrences in unlabeled data for task-agnostic representation learning, as exemplified by masked language model pretraining. In this chapter, we explore task-specific self-supervision, which leverages domain knowledge to automatically annotate noisy training examples for end applications, either by introducing labeling functions for annotating individual instances, or by imposing constraints over interdependent label decisions. We first present deep probabilistic logic (DPL), which offers a unifying framework for task-specific self-supervision by composing probabilistic logic with deep learning. DPL represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. Next, we present self-supervised self-supervision (S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial seed self-supervision, S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments on real-world applications such as biomedical machine reading and various text classification tasks show that task-specific self-supervision can effectively leverage domain expertise and often match the accuracy of supervised methods with a tiny fraction of human effort.

show abstract

Adaptive Rule Discovery for Labeling Text Data

Cited by 12 publications

References 11 publications

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

WRENCH: A Comprehensive Benchmark for Weak Supervision

Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning

Contact Info

Product

Resources

About