Training Complex Models with Multi-Task Weak Supervision

Abstract

As machine learning models continue to increase in complexity, collecting large hand-labeled training sets has become one of the biggest roadblocks in practice. Instead, weaker forms of supervision that provide noisier but cheaper labels are often used. However, these weak supervision sources have diverse and unknown accuracies, may output correlated labels, and may label different tasks or apply at different levels of granularity. We propose a framework for integrating and modeling such weak supervision sources by viewing them as labeling different related sub-tasks of a problem, which we refer to as the multi-task weak supervision setting. We show that by solving a matrix completion-style problem, we can recover the accuracies of these multi-task sources given their dependency structure, but without any labeled data, leading to higher-quality supervision for training an end model. Theoretically, we show that the generalization error of models trained with this approach improves with the number of unlabeled data points, and characterize the scaling with respect to the task and dependency structures. On three fine-grained classification problems, we show that our approach leads to average gains of 20.2 points in accuracy over a traditional supervised approach, 6.8 points over a majority vote baseline, and 4.1 points over a previously proposed weak supervision method that models tasks separately.

Learning to Denoise Distantly-Labeled Data for Entity Typing

“…For information extraction tasks such as relation extraction and entity typing, distant supervision (Mintz et al, 2009) is a powerful approach for adding more data, using a knowledge base (Del Corro et al, 2015;Rabinovich and Klein, 2017) or heuristics (Ratner et al, 2016;Hancock et al, 2018) to automatically label instances. One can treat this data just like any other supervised data, but it is noisy; more effective approaches employ specialized probabilistic models (Riedel et al, 2010;Ratner et al, 2018a), capturing its interaction with other supervision (Wang and Poon, 2018) or breaking down aspects of a task on which it is reliable (Ratner et al, 2018b). However, these approaches often require sophisticated probabilistic inference for training of the final model.…”

Mendelian Randomization with Instrumental Variable Synthesis (IVY)

“…Specifically, the conditional independence assumption made in IVY for the inclusion of individual SNPs is valid because the same assumption is made for methods employing allele scores (Sebastiani et al, 2012). We point out the possibility of extending beyond the assumption of conditional independence such as handling SNPs occurring physically upon the same chromosome which display some degree of genetic linkage (Ratner et al, 2018). Empirical validation of such an extension is left for future research.…”

“…In IVY, we also made use of such SNPs. We refer interested readers to (Ratner et al, 2018) for a discussion of scenarios such as handling dependencies among * 's. In this case, Ivy generalizes beyond the standard conditional independence assumption.…”

“…Specifically, we focus upon the identification and representation of instrumental variables by learning a valid and influential latent instrumental variable from candidate genetic variants. Our framework leverages recent work on learning and combining noisy labels (Natarajan, Dhillon, Ravikumar, & Tewari, 2018); (Ratner, De Sa, Wu, Selsam, & Ré, 2016;Ratner et al, 2018), with the goal of more accurate prediction and representation of population-level variance using individual-level data. Our approach, termed Instrumental Variable sYnthesis (IVY), is complementary to existing causal effect estimators, whether classical (the Wald estimator (Wald, 1940), the two-stage least squares approach (Angrist et al, 1996) ), robust (Bowden, Davey Smith, Haycock, & Burgess,…”

Cross-Modal Data Programming Enables Rapid Medical Machine Learning

“…Moreover, motivated by the increasing volume of digitized but unlabeled medical data, we evaluate the scaling of our approach with respect to additional unlabeled data, and find that performance consistently increases as more unlabeled data is collected in a way that is consistent with theoretical predictions. 33,34 In summary, cross-modal data programming can lower a substantial barrier to machine learning model development in medicine by serving as a fundamental new interface that reduces labeling time required from domain experts, thereby providing a stepping stone towards widespread adoption of machine learning models to provide positive, tangible clinical impact.…”