SimuRec: Workshop on Synthetic Data and Simulation Methods for Recommender Systems Research

Abstract

There is significant interest lately in using synthetic data and simulation infrastructures for various types of recommender systems research. However, there are not currently any clear best practices around how best to apply these methods. We proposed a workshop to bring together researchers and practitioners interested in simulating recommender systems and their data to discuss the state of the art of such research and the pressing open methodological questions. The workshop resulted in a report authored by the participants that documents currently-known best practices on which the group has consensus and lays out an agenda for further research over the next 3-5 years to fill in places where we currently lack the information needed to make methodological recommendations.

Distributionally-Informed Recommender System Evaluation

“…There is currently a body of ongoing work on simulation for recommender systems and related research [5,31,51,62], some of which is explicitly aimed at quantifying uncertainty [53]. The vision we propose will have a symbiotic relationship with this line of research: such simulations, as we have noted in Section 4.5, provide a source of uncertainty over which we may want to analyze the distribution of system behavior, and the metrics and techniques developed to enable rigorous and thorough evaluation that accounts for distributions of effects and benefits will be valuable for reporting the results of such simulations.…”

Synthetic Data-Based Simulators for Recommender Systems: A Survey

“…One of the most promising approaches to cope with the above-mentioned issues is the usage of synthetic data and the modeling and simulation (M&S) of interactions between users and RSs Ekstrand et al (2021); Bernardi et al (2021); Kiyohara et al (2021); Balog et al (2022). It is worth mentioning here that the analysis of search requests in Google Scholar conducted in Ekstrand et al (2021) shows that among the papers published from 2017 to 2021 and presented at worldclass conferences on the subject of RSs, about 27% of papers use synthetic data and the M&S or discuss their applications to the task. Indeed, synthetic data and M&S may be used for various purposes in connection to the above-mentioned issues, in particular,…”

“…• to supplement and/or replace real-world data in the RS training and testing process with its synthetic analogues in the simulated environment and to overcome the data insufficiency problem del Carmen et al (2017); Ekstrand et al (2021) and the necessity to perform complex, costly and risky online experiments Kiyohara et al (2021); Bernardi et al (2021);…”

“…This problem can be solved on the basis of methods for publishing company datasets without the risk of revealing confidential data Slokom (2018); Dankar and Ibrahim (2021), which have recently been actively proposed by the community of RS developers. One of the most promising approaches for it is the generation of synthetic data Slokom (2018); Ekstrand et al (2021). Thus, instead of the publication of real-world data, a company may share a synthetic dataset itself or a pre-trained simulation framework that is able to produce realistic synthetic datasets.…”

“…Furthermore, as in the area of other data mining tasks, indications exist of certain reproducibility problems in today's research practice connected with the M&S under consideration Balog et al (2022). In addition, the existing comparative studies of the methods and simulators for testing RSs are still few in number Ekstrand et al (2021). As a result, it is often impossible to reasonably determine the advantages of various modeling schemes, to learn how to properly validate models for the RSs analysis and how to adopt best practices in using such methods.…”