Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Abstract

Weak Supervision (WS) techniques allow users to efficiently create large training datasets by programmatically labeling data with heuristic sources of supervision. While the success of WS relies heavily on the provided labeling heuristics, the process of how these heuristics are created in practice has remained under-explored. In this work, we formalize the development process of labeling heuristics as an interactive procedure, built around the existing workflow where users draw ideas from a selected set of development data for designing the heuristic sources. With the formalism, we study two core problems of how to strategically select the development data to guide users in efficiently creating informative heuristics, and how to exploit the information within the development process to contextualize and better learn from the resultant heuristics (Figure 1). Building upon two novel methodologies that effectively tackle the respective problems considered, we present Nemo, an end-to-end interactive system that improves the overall productivity of WS learning pipeline by an average 20% (and up to 47% in one task) compared to the prevailing WS approach.

AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models

“…Ein-Dor et al ( 2020) offer an empirical study of active learning with PLMs. Very recently, there are also several works attempted to query labeling functions for weakly-supervised learning (Boecking et al, 2020;Hsieh et al, 2022;Zhang et al, 2022b). In our study, we leverage the power of unlabeled instances via self-training to further promote the performance of AL.…”

Understanding Programmatic Weak Supervision via Source-aware Influence Function

“…First, by understanding what are the deciding factors that lead to a model's specific prediction, users are able to verify whether such influence is desirable in terms of critical aspects such as model safety and fairness [11,35,16]. Secondly, since developing LFs is still inevitably a partly manual process in real-world applications and may involve human expert in an iterative process [5,20,48], it is beneficial to offer users with feedback of how individual component (like each LF) influences the end model performance on the downstream task. Finally, the rendered understanding of (in)efficacy of the PWS pipeline can be naturally exploited to automatically improve the end model performance.…”