A cross-benchmark comparison of 87 learning to rank methods

Tax, Niek; Bockting, Sander; Hiemstra, Djoerd

doi:10.1016/j.ipm.2015.07.002

Cited by 62 publications

(40 citation statements)

References 116 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We hypothesized that LambdaMART would be the best based on its superior performance in Web search, and the results indeed con rm this hypothesis; it achieves the highest test performance for each target objective, followed by AdaRank and RankNet. is observation is consistent with prior benchmark studies on web search dataset [30]. Linear classi cation based approaches such as L1 regularized Logistic Regression (L1LR) and L1 Regularized L2 loss SVM Classi er (L1L2SVM) also perform well.…”

Section: Comparison Of Letor Methodssupporting

confidence: 87%

“…Over the past decade, Learning to Rank (LETOR) methods, which involve applying machine learning techniques on ranking problems, have proven to be very successful in optimizing search engines; speci cally, they have been extensively studied in the context of Web search [3,7,18,30] to combine multiple features to optimize ranking. us, not surprisingly, learning to rank is also the backbone technique for optimizing the ranking of products in product search.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Application of Learning to Rank for E-Commerce Search

Santu

Sondhi

Zhai

2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

E-Commerce (E-Com) search is an emerging important new application of information retrieval. Learning to Rank (LETOR) is a general e ective strategy for optimizing search engines, and is thus also a key technology for E-Com search. While the use of LETOR for web search has been well studied, its use for E-Com search has not yet been well explored. In this paper, we discuss the practical challenges in applying learning to rank methods to E-Com search, including the challenges in feature representation, obtaining reliable relevance judgments, and optimally exploiting multiple user feedback signals such as click rates, add-to-cart ratios, order rates, and revenue. We study these new challenges using experiments on industry data sets and report several interesting ndings that can provide guidance on how to optimally apply LETOR to E-Com search: First, popularity-based features de ned solely on product items are very useful and LETOR methods were able to e ectively optimize their combination with relevance-based features. Second, query a ribute sparsity raises challenges for LETOR, and selecting features to reduce/avoid sparsity is bene cial. ird, while crowdsourcing is o en useful for obtaining relevance judgments for Web search, it does not work as well for E-Com search due to di culty in eliciting su ciently ne grained relevance judgments. Finally, among the multiple feedback signals, the order rate is found to be the most robust training objective, followed by click rate, while add-to-cart ratio seems least robust, suggesting that an e ective practical strategy may be to initially use click rates for training and gradually shi to using order rates as they become available.

show abstract

Section: Comparison Of Letor Methodssupporting

confidence: 87%

Section: Introductionmentioning

confidence: 99%

On Application of Learning to Rank for E-Commerce Search

Santu

Sondhi

Zhai

2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…The first way to extend our work is to do more experiments to cover better the parameter space of the problem of comparing supervised ML algorithms. That implies using more data sets where the notion of dominant algorithm can be extended [20], as well as trying all possible evaluation techniques. Another extension would be to vary the number of features and consider more algorithms.…”

Section: Discussionmentioning

confidence: 99%

Quality-efficiency trade-offs in machine learning for text processing

Baeza-Yates

Liaghat

2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Nowadays, the amount of available digital documents is rapidly growing from a variety of sources. Extracting information from these documents and finding useful information from such collections has become a challenge, which makes organizing and processing textual big data a necessity. Data mining, machine learning, and natural language processing are powerful techniques that can be used together to deal with this big challenge. Depending on the task or problem at hand, there are many different approaches that can be used. The methods available are continuously being optimized, but not all these methods have been tested and compared in a set of problems that can be solved using supervised machine learning algorithms. The question is what happens to the quality of methods if we increase the training data size from, say, 100 MB to over 1 GB? Moreover, are quality gains worth it when the rate of data processing diminishes? Can we trade quality for time efficiency and recover the quality loss by just being able to process more data?We attempt to answer these questions in a general way for text processing tasks, considering the trade-offs involving training data size, learning time, and quality obtained. Hence, we propose a performance trade-off framework and apply it to three important text processing problems: Named Entity Recognition, Sentiment Analysis, and Document Classification. These problems were also chosen because they have different levels of object granularity: words, paragraphs, and documents. For each problem, we selected several supervised machine learning algorithms and we evaluated the trade-offs of these different methods on large publicly available data sets (news, reviews, patents). We use different data subsets of increasing size ranging from 50 MB to several GB, to explore these trade-offs. For the two last problems, we consider similar algorithms with two different data sets and two different evaluation techniques, to study the impact of the data itself and the evaluation technique on the resulting trade-offs. We find that the results do not change significantly and that most of the time the best algorithms are the ones with fastest processing time. However, we also show that the results for small data (say less than 100 MB) are different from the results for big data and in those cases the best algorithm is much harder to determine.

show abstract

“…To evaluate the overall performance of activity filtering techniques, we use the number of other filtering techniques that it can beat over all the seventeen event logs of Table 2. This metric, known as winning number, is commonly used for evaluation in the Information Retrieval (IR) field [32,37]. Formally, winning number is defined as…”

Section: Aggregated Analysis Over All Event Logsmentioning

confidence: 99%

Discovering more precise process models from event logs by filtering out chaotic activities

2018

Self Cite

View full text Add to dashboard Cite

Process Discovery is concerned with the automatic generation of a process model that describes a business process from execution data of that business process. Real life event logs can contain chaotic activities. These activities are independent of the state of the process and can, therefore, happen at rather arbitrary points in time. We show that the presence of such chaotic activities in an event log heavily impacts the quality of the process models that can be discovered with process discovery techniques. The current modus operandi for filtering activities from event logs is to simply filter out infrequent activities. We show that frequency-based filtering of activities does not solve the problems that are caused by chaotic activities. Moreover, we propose a novel technique to filter out chaotic activities from event logs. We evaluate this technique on a collection of seventeen reallife event logs that originate from both the business process management domain and the smart home environment domain. As demonstrated, the developed activity filtering methods enable the discovery of process models that are more behaviorally specific compared to process models that are discovered using standard frequency-based filtering.

show abstract

A cross-benchmark comparison of 87 learning to rank methods

Cited by 62 publications

References 116 publications

On Application of Learning to Rank for E-Commerce Search

On Application of Learning to Rank for E-Commerce Search

Quality-efficiency trade-offs in machine learning for text processing

Discovering more precise process models from event logs by filtering out chaotic activities

Contact Info

Product

Resources

About