A Two‐stage active learning method for learning to rank

Abstract

Learning to rank (L2R) algorithms use a labeled training set to generate a ranking model that can later be used to rank new query results. These training sets are costly and laborious to produce, requiring human annotators to assess the relevance or order of the documents in relation to a query. Active learning algorithms are able to reduce the labeling effort by selectively sampling an unlabeled set and choosing data instances that maximize a learning function's effectiveness. In this article, we propose a novel two-stage active learning method for L2R that combines and exploits interesting properties of its constituent parts, thus being effective and practical. In the first stage, an association rule active sampling algorithm is used to select a very small but effective initial training set. In the second stage, a query-bycommittee strategy trained with the first-stage set is used to iteratively select more examples until a preset labeling budget is met or a target effectiveness is achieved. We test our method with various LETOR benchmarking data sets and compare it with several baselines to show that it achieves good results using only a small portion of the original training sets.

A Practical and Effective Sampling Selection Strategy for Large Scale Deduplication

“…We configured PPJoin++ with Jaccard as the similarity function. We also used the active sampling implementation (SSAR) described in [21].…”

“…The second stage of T3S aims at incrementally removing the non-informative or redundant pairs inside each sample level by using the SSAR (Selective Sampling using Association Rules) active learning method [21]. By redundant, we mean pairs carrying very similar information; the inclusion of a redundant pair in the training set for the classification step does not contribute with useful information for the learning process.…”

“…1(2)). In the second stage, the redundant information that is selected in the subsamples is removed by means of a rule-based active sampling [21] (Fig. 1(3)) which requires no previously labeled training set.…”

Leveraging active learning to reduce human effort in the generation of ground‐truth for entity resolution

“…In these strategies, ML algorithms, in conjunction with a human expert, seek to select informative data for training, resulting in the generation of smaller balanced labeled sets, but sufficient to train accurate algorithms . In the scope of AL, informative data is those which, without presenting redundancy between them, are capable of representing the characteristics of the dataset …”

“…The selection of data is performed through the application of heuristics, such as AL based on sample by uncertainty or consultation of a committee of classifiers . The set T , resulting from the selection, can be used to train ML algorithms as accurately as those algorithms that make use of large training sets generated by random sampling …”