Offline recommender system evaluation: Challenges and new directions

Castells, Pablo; Moffat, Alistair

doi:10.1002/aaai.12051

Cited by 38 publications

(14 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 Supplemental material, including source code, scripts, datasets, and results are permanently placed in: https://github.com/recsyspolimi/recsys-2022-evaluation-of-recsyswith-impressions 2 Further studies may select different evaluation methodologies depending on their context, see Castells and Moffat [6].…”

Section: Experimental Methodologymentioning

confidence: 99%

“…However, this is rapidly changing as more impression datasets have been published and competitions encouraged their use. For instance: (i) interest in impressions research and use has risen [6,22], (ii) several RS challenges included impressions in their datasets, (iii) industries have presented case studies highlighting the effects of impressions data in their recommendations [4,19], and (iv) three open datasets were recently-published ContentWise Impressions [26], MIND [32], and FINN.no Slates [10,11]. Due to the limited use of impressions in RS, there exist several open research questions, e.g., whether to evaluate recommendation models with impressions data requires to develop a specific methodology as is the case in other scenarios [12,14], characterization of signals and biases in impressions, and challenges regarding the use of impressions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards the Evaluation of Recommender Systems with Impressions

Maurera

Dacrema

Cremonesi

2022

Proceedings of the 16th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

In Recommender Systems, impressions are a relatively new type of information that records all products previously shown to the users. They are also a complex source of information, combining the effects of the recommender system that generated them, search results, or business rules that may select specific products for recommendations. The fact that the user interacted with a specific item given a list of recommended ones may benefit from a richer interaction signal, in which some items the user did not interact with may be considered negative interactions. This work presents a preliminary evaluation of recommendation models with impressions. First, impressions are characterized by describing their assumptions, signals, and challenges. Then, an evaluation study with impressions is described. The study's goal is two-fold: to measure the effects of impressions data on properly-tuned recommendation models using current open-source datasets and disentangle the signals within impressions data. Preliminary results suggest that impressions data and signals are nuanced, complex, and effective at improving the recommendation quality of recommenders. This work publishes the source code, datasets, and scripts used in the evaluation to promote reproducibility in the domain.

show abstract

Section: Experimental Methodologymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Towards the Evaluation of Recommender Systems with Impressions

Maurera

Dacrema

Cremonesi

2022

Proceedings of the 16th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

show abstract

“…Evaluating recommendation systems is a crucial stage in validating their usage. Consequently, several studies have investigated the impact of time on online evaluations [31] [32] [33] because this temporal factor aids in interpreting the acceptance of the obtained recommendation results [34]. Online evaluation is the best method to evaluate RSs and CARS.…”

Section: Literature Reviewmentioning

confidence: 99%

Toward a Systematic Evaluation Approach of Point Of Interest Recommendation Algorithms of a Novel Smart Tourism Tool

HADJHENNI,

DENNOUNI,

SLAMA

2024

IJCDS

View full text Add to dashboard Cite

Intelligent tourism can increase the interest of nomadic tourists in discovering new cities. However, many Points of Interest (POIs) are available, creating an information overload for tourists when choosing POIs to visit. For this reason, CARS (Context-Aware Recommendation Systems) can play an important role by exploiting the experiences of previous tourists and their contexts to recommend attractive POIs. Consequently, choosing the right POI recommendation algorithm (RA) for CARS is crucial because it involves the costly intervention of real tourists during the test phase. In order to make this phase more cost-effective, we can test several RAs simultaneously in order to assess their limitations in terms of cold start and tourist satisfaction. To compare these RAs, we propose in this article an approach called SEPRA (Systematic Evaluation for POI Recommendation Algorithms), which allows us to carry out an initial online evaluation of each tourist during their visit and a second offline evaluation of each CARS after the end of each POI path. To achieve this objective, we designed and implemented a new smart tourism tool that makes POI recommendations using two algorithms: the first is based on tourist/tourist similarity, and the second uses POI/POI similarity. These algorithms use memory-based collaborative filtering and are executed in parallel by our tool in the form of CARSs, incorporating time or weather as context variables. To evaluate these systems during their test phases, Our approach enables: (1) the calculation of prediction accuracy; (2) the examination of the relevance of the recommended POIs; and (3) the estimation of the acceptance rate of the recommendation process. Finally, the experimental results obtained with our approach show that the algorithm based on tourist similarity is more resistant to the cold start problem during the test phase and has a better satisfaction rate than the algorithm based on POI similarity.

show abstract

“…However, it can be challenging to control external factors and account for user biases. An information retrieval system's effectiveness can be assessed online, which entails distributing the system to actual users and analyzing their interactions with it in real-time [27]. There is not much information on how deploying a hybrid recommender system relates to the evaluation.…”

Section: ) Offline Evaluationmentioning

confidence: 99%

Exploring the Impact of Hybrid Recommender Systems on Personalized Mental Health Recommendations

Mazlan,

Abdullah,

Ahmad

2023

IJACSA

View full text Add to dashboard Cite

Personalized mental health recommendations are crucial in addressing the diverse needs and preferences of individuals seeking mental health support. This research aims to study the investigates the impact of hybrid recommender systems on the provision of personalized recommendations for mental health interventions. This paper explores the integration of various recommendation techniques, including collaborative filtering, content-based filtering, and knowledge-based filtering, within the hybrid system to leverage their respective strengths for Personalized Mental Health Recommendations. Additionally, this paper discusses the challenges and considerations involved in combining multiple techniques, such as data integration and algorithm selection for Hybrid Recommender System for this domain. Furthermore, this paper also discusses the data sources that are typically used in hybrid recommender systems for mental health and evaluation metrics that are employed to assess the effectiveness of the hybrid recommender system. Future research opportunities, including incorporating emerging technologies and leveraging novel data sources, are identified to further enhance the performance and relevance of hybrid recommender systems in the mental health domain. The findings of this research contribute to the advancement of personalized mental health support and the development of effective recommendation systems tailored to individual mental health needs.

show abstract

Offline recommender system evaluation: Challenges and new directions

Cited by 38 publications

References 75 publications

Towards the Evaluation of Recommender Systems with Impressions

Towards the Evaluation of Recommender Systems with Impressions

Toward a Systematic Evaluation Approach of Point Of Interest Recommendation Algorithms of a Novel Smart Tourism Tool

Exploring the Impact of Hybrid Recommender Systems on Personalized Mental Health Recommendations

Contact Info

Product

Resources

About