A Selective Sampling Strategy for Label Ranking

Amini, Massih R.; Usunier, Nicolas; Laviolette, François; Lacasse, Alexandre; Gallinari, Patrick

doi:10.1007/11871842_7

Cited by 7 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Active learning has been actively extended to rank learning [2,15,16,28,45]. Based on uncertainty sampling, [45] selected the most ambiguous document pairs, in which two documents received close scores predicted by the current model, as informative examples.…”

Section: Active Rank Learningmentioning

confidence: 99%

“…target domain) where no or just a few labeled data is available. Active rank learning [2,15,16,28,45] is motivated to proactively select and annotate those most informative set of examples, which are expected that, if labeled, they can maximize the information value to the ranking function.…”

Section: Introductionmentioning

confidence: 99%

“…For this purpose, we need to conquer two obvious challenges: (1) how to effectively bridge the shift of data across domains in terms of different joint distributions of rank features and rank labels; (2) how to select the minimum set of target-domain examples to label that can well represent the ranking information of the target domain. In this work, we propose to combine the utilization of relevant knowledge in sourcedomain training data and the active selection of informative queries from target domain, where the performance of these two components can be mutually reinforced within a unified iterative framework.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Relevant knowledge helps in choosing right teacher

Cai

Gao

Zhou

et al. 2011

Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.

show abstract

Section: Active Rank Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Relevant knowledge helps in choosing right teacher

Cai

Gao

Zhou

et al. 2011

Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…We tested the performance of our method (denoted by LossMin) against four baselines: the entropy-based sampling method of [22] (denoted by Entropy), the uncertainty sampling heuristic of [30] (denoted by Uncertain), the divergencebased sampling strategy of [1] (denoted by Diverse), and random sampling (denoted by Random). Entropy method [22] samples the most confusing instances for the current ranker which are identified via estimating the bipartite ranking error [8] that counts an error each time a relevant instance is ranked lower than an irrelevant one.…”

Section: Data and Problem Setupmentioning

confidence: 99%

Active Sampling for Rank Learning via Optimizing the Area under the ROC Curve

Dönmez

Carbonell

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learning ranking functions is crucial for solving many problems, ranging from document retrieval to building recommendation systems based on an individual user's preferences or on collaborative filtering. Learning-to-rank is particularly necessary for adaptive or personalizable tasks, including email prioritization, individualized recommendation systems, personalized news clipping services and so on. Whereas the learning-to-rank challenge has been addressed in the literature, little work has been done in an active-learning framework, where requisite user feedback is minimized by selecting only the most informative instances to train the rank learner. This paper addresses active rank-learning head on, proposing a new sampling strategy based on minimizing hinge rank loss, and demonstrating the effectiveness of the active sampling method for rankSVM on two standard rank-learning datasets. The proposed method shows convincing results in optimizing three performance metrics, as well as improvement against four baselines including entropybased, divergence-based, uncertainty-based and random sampling methods.

show abstract

“…Another recent development in active rank learning is the divergence-based sampling method of (Amini et al, 2006). The proposed method selects the samples at which two different ranking functions maximally disagree.…”

Section: Related Workmentioning

confidence: 99%

Optimizing Estimated Loss Reduction for Active Sampling in Rank Learning

Dönmez¹,

Carbonell²

2008

View full text Add to dashboard Cite

Learning to rank is becoming an increasingly popular research area in machine learning. The ranking problem aims to induce an ordering or preference relations among a set of instances in the input space. However, collecting labeled data is growing into a burden in many rank applications since labeling requires eliciting the relative ordering over the set of alternatives. In this paper, we propose a novel active learning framework for SVM-based and boosting-based rank learning. Our approach suggests sampling based on maximizing the estimated loss differential over unlabeled data. Experimental results on two benchmark corpora show that the proposed model substantially reduces the labeling effort, and achieves superior performance rapidly with as much as 30% relative improvement over the margin-based sampling baseline.

show abstract

A Selective Sampling Strategy for Label Ranking

Cited by 7 publications

References 9 publications

Relevant knowledge helps in choosing right teacher

Relevant knowledge helps in choosing right teacher

Active Sampling for Rank Learning via Optimizing the Area under the ROC Curve

Optimizing Estimated Loss Reduction for Active Sampling in Rank Learning

Contact Info

Product

Resources

About