Learning from a mixture of labeled and unlabeled examples with parametric side information

Ratsaby, Joel; Venkatesh, Santosh S.

doi:10.1145/225298.225348

Cited by 65 publications

(48 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As mentioned in Section 1, a typical assumption in a generative setting is that D is a mixture with the probability density function p(x|θ) = p 0 · p 0 (x|θ 0 ) + p 1 · p 1 (x|θ 1 ) (see for instance [Ratsaby and Venkatesh 1995;Castelli and Cover 1995;). In other words, the labeled examples are generated according to the following · 29 mechanism: a label y ∈ {0, 1} is drawn according to the distribution of classes {p 0 , p 1 } and then a corresponding random feature vector is drawn according to the class-conditional density p y .…”

Section: Connections To Generative Modelsmentioning

confidence: 99%

“…Essentially once the decision border is estimated, a small labeled sample suffices to learn (with high confidence and small error) the appropriate class labels associated with the two disjoint regions generated by the estimate of the Bayes decision border. To see how we can incorporate this setting in our model, consider for illustration the setting in [Ratsaby and Venkatesh 1995]; there they assume that p 0 = p 1 , and that the class conditional densities are d-dimensional Gaussians with unit covariance and unknown mean vectors θ i ∈ R d . The algorithm used is the following: the unknown parameter vector θ = (θ 0 , θ 1 ) is estimated from unlabeled data using a maximum likelihood estimate; this determines a hypothesis which is a linear separator that passes through the point (θ 0 +θ 1 )/2 and is orthogonal to the vectorθ 1 −θ 0 ; finally each of the two decision regions separated by the hyperplane is labeled according to the majority of the labeled examples in the region.…”

Section: Connections To Generative Modelsmentioning

confidence: 99%

“…Specifically, we can identify a legal hypothesis f θ with the set of parameters θ = (θ 0 , θ 1 ) that determine it, and then we can define χ(f θ , D) = E x∈D [log(p(x|θ))]. [Ratsaby and Venkatesh 1995] show that if the unlabeled sample is large enough, then all hypotheses specified by parameters θ which are close enough to θ, will have the property that their empirical compatibilities will be close enough to their true compatibilities. This then implies (together with other observations about Gaussian mixtures) that the maximum likelihood estimate will be close enough to θ, up to permutations.…”

Section: Connections To Generative Modelsmentioning

confidence: 99%

See 2 more Smart Citations

A discriminative model for semi-supervised learning

Balcan

Blum

2010

J. ACM

100

View full text Add to dashboard Cite

Supervised learning -that is, learning from labeled examples -is an area of Machine Learning that has reached substantial maturity. It has generated general-purpose and practically-successful algorithms and the foundations are quite well understood and captured by theoretical frameworks such as the PAC-learning model and the Statistical Learning theory framework. However, for many contemporary practical problems such as classifying web pages or detecting spam, there is often additional information available in the form of unlabeled data, which is often much cheaper and more plentiful than labeled data. As a consequence, there has recently been substantial interest in semi-supervised learning -using unlabeled data together with labeled data -since any useful information that reduces the amount of labeled data needed can be a significant benefit. Several techniques have been developed for doing this, along with experimental results on a variety of different learning problems. Unfortunately, the standard learning frameworks for reasoning about supervised learning do not capture the key aspects and the assumptions underlying these semisupervised learning methods.In this paper we describe an augmented version of the PAC model designed for semi-supervised learning, that can be used to reason about many of the different approaches taken over the past decade in the Machine Learning community. This model provides a unified framework for analyzing when and why unlabeled data can help, in which one can analyze both sample-complexity and algorithmic issues. The model can be viewed as an extension of the standard PAC model where, in addition to a concept class C, one also proposes a compatibility notion: a type of compatibility that one believes the target concept should have with the underlying distribution of data. Unlabeled data is then potentially helpful in this setting because it allows one to estimate compatibility over the space of hypotheses, and to reduce the size of the search space from the whole set of hypotheses C down to those that, according to one's assumptions, are a-priori reasonable with respect to the distribution. As we show, many of the assumptions underlying existing semi-supervised learning algorithms can be formulated in this framework.After proposing the model, we then analyze sample-complexity issues in this setting: that is, how much of each type of data one should expect to need in order to learn well, and what the key quantities are that these numbers depend on. We also consider the algorithmic question of how to efficiently optimize for natural classes and compatibility notions, and provide several algorithmic results including an improved bound for Co-Training with linear separators when the distribution satisfies independence given the label.

show abstract

Section: Connections To Generative Modelsmentioning

confidence: 99%

Section: Connections To Generative Modelsmentioning

confidence: 99%

Section: Connections To Generative Modelsmentioning

confidence: 99%

See 1 more Smart Citation

A discriminative model for semi-supervised learning

Balcan

Blum

2010

J. ACM

100

View full text Add to dashboard Cite

show abstract

“…For instance, we could search for pages pointed to with links having the phrase \my advisor" and use them as \probably positive" examples to further train a learning algorithm based on the words on the text page, and vice-versa. We call this type of bootstrapping cotraining, and it has a close connection to bootstrapping from incomplete data in the Expectation-Maximization setting see, for instance, 7,15]. The question this raises is: is there any reason to believe co-training will help?…”

Section: Introductionmentioning

confidence: 99%

Combining labeled and unlabeled data with co-training

Blum

Mitchell

1998

Proceedings of the Eleventh Annual Conference on Computational Learning Theory

4,415

3,213

View full text Add to dashboard Cite

We consider the problem of using a large unlabeled sample to boost performance of a learning algorithm when only a small set of labeled examples is available. In particular, we consider a setting in which the description of each example can be partitioned into two distinct views, motivated by the task of learning to classify web pages. For example, the description of a web page can be partitioned into the words occurring on that page, and the words occurring in hyperlinks that point to that page. We assume that either view of the example would be su cient for learning if we had enough labeled data, but our goal is to use both views together to allow inexpensive unlabeled data to augment a m uch smaller set of labeled examples. Speci cally, the presence of two distinct views of each example suggests strategies in which t wo learning algorithms are trained separately on each view, and then each algorithm's predictions on new unlabeled examples are used to enlarge the training set of the other. Our goal in this paper is to provide a PAC-style analysis for this setting, and, more broadly, a P AC-style framework for the general problem of learning from both labeled and unlabeled data. We also provide empirical results on real web-page data indicating that this use of unlabeled examples can lead to signi cant improvement o f h ypotheses in practice. As part of our analysis, we p r o vide new reThis is an extended version of a paper that appeared in the

show abstract

“…A substantial body of literature has investigated unlabeled data in the context of supervised learning, although not in the same way we have considered in this paper. Most work in this area adopts the perspective of parametric probability modeling and uses unlabeled data as part of a maximum likelihood (EM) or discriminative training procedure (Miller & Uyar, 1997;Castelli & Cover, 1996;Ratsaby & Venkatesh, 1995;Gutfinger & Sklansky, 1991;O'Neill, 1978). Another common idea is to supply artificial labels to unlabeled examples and use this data directly in a supervised training procedure (Blum & Mitchell, 1998;Towell, 1996).…”

Section: Resultsmentioning

confidence: 99%

Untitled

Schuurmans

Southey

2002

Machine Learning

View full text Add to dashboard Cite

Abstract. We present a general approach to model selection and regularization that exploits unlabeled data to adaptively control hypothesis complexity in supervised learning tasks. The idea is to impose a metric structure on hypotheses by determining the discrepancy between their predictions across the distribution of unlabeled data. We show how this metric can be used to detect untrustworthy training error estimates, and devise novel model selection strategies that exhibit theoretical guarantees against over-fitting (while still avoiding under-fitting). We then extend the approach to derive a general training criterion for supervised learning-yielding an adaptive regularization method that uses unlabeled data to automatically set regularization parameters. This new criterion adjusts its regularization level to the specific set of training data received, and performs well on a variety of regression and conditional density estimation tasks. The only proviso for these methods is that sufficient unlabeled training data be available.

show abstract

Learning from a mixture of labeled and unlabeled examples with parametric side information

Cited by 65 publications

References 9 publications

A discriminative model for semi-supervised learning

A discriminative model for semi-supervised learning

Combining labeled and unlabeled data with co-training

Untitled

Contact Info

Product

Resources

About