Robust Data Programming with Precision-guided Labeling Functions

Abstract

Scarcity of labeled data is a bottleneck for supervised learning models. A paradigm that has evolved for dealing with this problem is data programming. An existing data programming paradigm allows human supervision to be provided as a set of discrete labeling functions (LF) that output possibly noisy labels to input instances and a generative model for consolidating the weak labels. We enhance and generalize this paradigm by supporting functions that output a continuous score (instead of a hard label) that noisily correlates with labels. We show across five applications that continuous LFs are more natural to program and lead to improved recall. We also show that accuracy of existing generative models is unstable with respect to initialization, training epochs, and learning rates. We give control to the data programmer to guide the training process by providing intuitive quality guides with each LF. We propose an elegant method of incorporating these guides into the generative model. Our overall method, called CAGE, makes the data programming paradigm more reliable than other tricks based on initialization, sign-penalties, or soft-accuracy constraints.

Semi-Supervised Data Programming with Subset Selection

“…Data Programming and Unsupervised Learning: Snorkel (Ratner et al, 2016) has been proposed as a generative model to determine correct label probability using consensus on the noisy and conflicting labels assigned by the discrete LFs. Chatterjee et al (2020) proposed a graphical model, CAGE, that uses continuous-valued LFs with scores obtained using soft match techniques such as cosine similarity of word vectors, TF-IDF score, distance among entity pairs, etc. Owing to its generative model, Snorkel is highly sensitive to initialisation and hyper-parameters.…”

“…may also produce conflicting labels. In the past, generative models such as Snorkel (Ratner et al, 2016) and CAGE (Chatterjee et al, 2020) have been proposed for consensus on the noisy and conflicting labels assigned by the discrete LFs to determine the probability of the correct labels. Labels thus obtained could be used for training any supervised model/classifier and evaluated on a test set.…”

“…It acts as a form of semi-supervision by trying to increase the confidence of the predictions made by the model on the unlabelled dataset. Third Component (L3): The third component L CE P f φ (y|x i ), g(l i ) is the cross-entropy of the classification model using the hypothesised labels from CAGE (Chatterjee et al, 2020) on U. Given that l i is the output vector of all labelling functions for any x i ∈ U, we specify the predicted label for x i using the LF-based graphical model P θ (l i , y) from eqn.…”

“…Sixth Component (L6): The sixth component KL(P f φ (y|x i ), P θ (y|l i )) is the Kullback-Leibler (KL) divergence between the predictions of both the models, viz., feature-based model f φ and the LF-based graphical model P θ summed over every example x i ∈ U ∪ L. Through this term, we try and make the models agree in their predictions over the union of the labelled and unlabelled datasets. Quality Guides (QG): As the last component in our objective, we use quality guides R(θ|{q j }) on LFs, which have been shown in (Chatterjee et al, 2020) to stabilise the unsupervised likelihood training while using labelling functions. Let q j be the fraction of cases where λ j correctly triggered, and let q t j be the user's belief on the fraction of examples x i where y i and l ij agree.…”

Lighten

“…Firstly, avoiding over-dependence on such hand-crafted features, since the above approaches limit the scope for in-the-wild HOI detections. Such over-dependence has been averted in both textual [2] and image [18] domains and we take inspiration from such works. More often than not, the 3D poses or 3D centroids of objects (used as features) are either not available or are too erroneously estimated to be simply plugged into a model trained on hand-crafted features.…”