ERR.Rank: An algorithm based on learning to rank for direct optimization of Expected Reciprocal Rank

Ghanbari, Elham; Shakery, Azadeh

doi:10.1007/s10489-018-1330-z

Cited by 8 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The metrics we utilized for evaluation were; Mean Reciprocal Rank (MRR) (Ghanbari & Shakery, 2019), Precision at 5 (P@5) (Sharma et al, 2020), and Normalized Discounted Cumulative Gain at 5 (NCDG@5) (Alqahtani et al, 2020). A threshold was used for the 5 best matches and the top match was given as the output.…”

Section: Machine Learning Technique and Interpreting Resultsmentioning

confidence: 99%

Detecting fake news and disinformation using artificial intelligence and machine learning to avoid supply chain disruptions

et al. 2022

View full text Add to dashboard Cite

Fake news and disinformation (FNaD) are increasingly being circulated through various online and social networking platforms, causing widespread disruptions and influencing decision-making perceptions. Despite the growing importance of detecting fake news in politics, relatively limited research efforts have been made to develop artificial intelligence (AI) and machine learning (ML) oriented FNaD detection models suited to minimize supply chain disruptions (SCDs). Using a combination of AI and ML, and case studies based on data collected from Indonesia, Malaysia, and Pakistan, we developed a FNaD detection model aimed at preventing SCDs. This model based on multiple data sources has shown evidence of its effectiveness in managerial decision-making. Our study further contributes to the supply chain and AI-ML literature, provides practical insights, and points to future research directions.

show abstract

Section: Machine Learning Technique and Interpreting Resultsmentioning

confidence: 99%

Detecting fake news and disinformation using artificial intelligence and machine learning to avoid supply chain disruptions

et al. 2022

View full text Add to dashboard Cite

show abstract

“…(2.1) to get the ranking position by pairwise comparison, which is closer in nature to sorting. Taking all the above into account, and following the same rationale as in Sections 2.3.1 and 2.3.3, we do not contemplate more complex techniques such as Boosting [127,40] or multi-agent learning [152], so that metrics are compared on a base recommendation model derived from Top-N-Rank. Specifically, we replace ReLU by a sigmoid function and approximate the ranking position as…”

Section: Listwise Metric Optimizationmentioning

confidence: 99%

Metric Optimization and Mainstream Bias Mitigation in Recommender Systems

2023

SIGIR Forum

View full text Add to dashboard Cite

Recommender Systems have drawn extensive attention in recent decades. The most important factor to make a recommender succeed is user satisfaction, which is largely reflected by the recommendation accuracy. In accordance, this thesis dives into recommendation accuracy from two different perspectives that, unfortunately, are at tension with each other: achieving the maximum overall recommendation accuracy, and balancing that accuracy among all users. The first part of this thesis focuses on maximizing the overall recommendation accuracy Li et al. [2021a]. This accuracy is usually evaluated with some user-oriented metrics tailored to the recommendation scenario. However, recommendation models could be trained to maximize some other generic criteria that do not necessarily align with the criteria ultimately captured by the metric above. Recent research usually assumes that the metric used for evaluation is also the metric used for training. We challenge this assumption, mainly because some metrics are more informative than others. Indeed, we show that models trained via the optimization of a loss inspired by Rank-Biased Precision (RBP) tend to yield higher accuracy, even when accuracy is measured with metrics other than RBP. However, the superiority of this RBP-inspired loss stems from further benefiting users who are already well-served, rather than helping those who are not. This observation inspires the second part of this thesis, where our focus turns to helping nonmainstream users, who are difficult to recommend to either because of the lack of data for effective modeling or a niche taste that is hard to match similar users. Our first effort consists in using side data, beyond the user-item interaction matrix, so that users and items are better represented. This will be of benefit especially for the non-mainstream users, for which the user-item matrix alone is ineffective. We propose Neural AutoEncoder Collaborative Filtering (NAECF) Li et al. [2021b], an adversarial learning architecture that, in addition to maximizing the recommendation accuracy, leverages side data to preserve the intrinsic properties of users and items and show that NAECF leads to better recommendations specially for non-mainstream users, while at the same time there is a marginal loss for the mainstream ones. Our second effort Li et al. [2023] consists in explicitly focusing more on non-mainstream users in a recommendation model. In particular, we propose a mechanism based on cost-sensitive learning that weighs users according to their mainstreamness, so that they get more attention during training. The result is a recommendation model tailored to non-mainstream users, that narrows the accuracy gap, and again at negligible cost to the mainstream users. Awarded by : Delft University of Technology, Delft, The Netherlands on 14 November 2023. Supervised by : Alan Hanjalic and Julián Urbano. Available at : https://doi.org/10.4233/uuid:235a5f5c-5e18-4e67-a8e6-84a7e310db12.

show abstract

“…(1) to get the ranking position by pairwise comparison, which is closer in nature to sorting. Taking all the above into account, and following the same rationale as in Sections 3.1 and 3.3, we do not contemplate more complex techniques such as Boosting [15,45] or multi-agent learning [52], so that metrics are compared on a base recommendation model derived from Top-N-Rank. Specifically, we replace ReLU by a sigmoid function and approximate the ranking position as…”

Section: Listwise Metric Optimizationmentioning

confidence: 99%

New Insights into Metric Optimization for Ranking-based Recommendation

Urbano

Hanjalic

2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Direct optimization of IR metrics has often been adopted as an approach to devise and develop ranking-based recommender systems. Most methods following this approach (e.g. TFMAP, CLiMF, Top-N-Rank) aim at optimizing the same metric being used for evaluation, under the assumption that this will lead to the best performance. A number of studies of this practice bring this assumption, however, into question. In this paper, we dig deeper into this issue in order to learn more about the effects of the choice of the metric to optimize on the performance of a ranking-based recommender system. We present an extensive experimental study conducted on different datasets in both pairwise and listwise learning-to-rank (LTR) scenarios, to compare the relative merit of four popular IR metrics, namely , , and , when used for optimization and assessment of recommender systems in various combinations. For the first three, we follow the practice of loss function formulation available in literature. For the fourth one, we propose novel loss functions inspired by for both the pairwise and listwise scenario. Our results confirm that the best performance is indeed not necessarily achieved when optimizing the same metric being used for evaluation. In fact, we find that -inspired losses perform at least as well as other metrics in a consistent way, and offer clear benefits in several cases. Interesting to see is that -inspired losses, while improving the recommendation performance for all uses, may lead to an individual performance gain that is correlated with the activity level of a user in interacting with items. The more active the users, the more they benefit. Overall, our results challenge the assumption behind the current research practice of optimizing and evaluating the same metric, and point to -based optimization instead as a promising alternative when learning to rank in the recommendation context. CCS CONCEPTS• Information systems → Recommender systems; Learning to rank; • General and reference → Metrics.

show abstract

ERR.Rank: An algorithm based on learning to rank for direct optimization of Expected Reciprocal Rank

Cited by 8 publications

References 32 publications

Detecting fake news and disinformation using artificial intelligence and machine learning to avoid supply chain disruptions

Detecting fake news and disinformation using artificial intelligence and machine learning to avoid supply chain disruptions

Metric Optimization and Mainstream Bias Mitigation in Recommender Systems

New Insights into Metric Optimization for Ranking-based Recommendation

Contact Info

Product

Resources

About