Towards the Evaluation of Recommender Systems with Impressions

Abstract

In Recommender Systems, impressions are a relatively new type of information that records all products previously shown to the users. They are also a complex source of information, combining the effects of the recommender system that generated them, search results, or business rules that may select specific products for recommendations. The fact that the user interacted with a specific item given a list of recommended ones may benefit from a richer interaction signal, in which some items the user did not interact with may be considered negative interactions. This work presents a preliminary evaluation of recommendation models with impressions. First, impressions are characterized by describing their assumptions, signals, and challenges. Then, an evaluation study with impressions is described. The study's goal is two-fold: to measure the effects of impressions data on properly-tuned recommendation models using current open-source datasets and disentangle the signals within impressions data. Preliminary results suggest that impressions data and signals are nuanced, complex, and effective at improving the recommendation quality of recommenders. This work publishes the source code, datasets, and scripts used in the evaluation to promote reproducibility in the domain.

Workshop on Learning and Evaluating Recommendations with Impressions (LERI)

“…Recently, a source of information that was previously almost unavailable to the wider research community has emerged with the potential to impact the field in numerous ways: impressions. Impressions [7,15,25,28,37] refer to the items displayed on the screen when a user interacts (or not) with them and are the product of the whole recommendation engine [7,21,22]. Impressions constitute a nuanced and intricate data source that raises novel research questions, opportunities, and challenges.…”

“…Among the new research opportunities opened by impressions, being able to distinguish between the items that the user observed and did not observe could allow to provide better assumptions on how to label missing interactions. Some studies consider impressions to be a positive interaction signal, while others view them as negative signals [21].…”