Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems

Winecoff, Amy A.; Sun, Matthew Z.; Lucherini, Eli; Narayanan, Arvind

doi:10.48550/arxiv.2107.14333

Cited by 2 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such cases have been reported e.g. in Dacrema et al (2021); Winecoff et al (2021). For example, Winecoff et al (2021) discusses four papers on the study of RS filter bubble effects which came to conflicting conclusions on the effect mechanism.…”

Section: Simulator's Approbation Reviewmentioning

confidence: 89%

“…This block of criteria is motivated by modern requirements on the reproducibility of research in the field of RSs development. In particular, we include these criteria in order to follow the general line of Dacrema et al (2021); Winecoff et al (2021) about the need for careful analysis of the interpretability and reproducibility in the field. In this regard, we distinguish simulators by that they have:…”

Section: Simulator's Approbationmentioning

confidence: 99%

“…in Dacrema et al (2021); Winecoff et al (2021). For example, Winecoff et al (2021) discusses four papers on the study of RS filter bubble effects which came to conflicting conclusions on the effect mechanism. It turns out that the studies have not agreed upon scientific terms and this led to heterogeneous and sometimes contradictory results.…”

Section: Simulator's Approbation Reviewmentioning

confidence: 99%

“…It turns out that the studies have not agreed upon scientific terms and this led to heterogeneous and sometimes contradictory results. Based on the example, the authors of Winecoff et al (2021) argue that the overall situation in the field results in a "patchwork of theoretical motivations, approaches, and implementations that are difficult to reconcile." Furthermore, the "lack of consensus on best practices in simulation studies of RS has limited their potential scientific impact".…”

Section: Simulator's Approbation Reviewmentioning

confidence: 99%

“…In spite of the great interest towards the M&S of interactions between users and RSs in both academic and industrial spheres, the base ideas of these methods, as well as their implementation and application methodology, seem to vary significantly Winecoff et al (2021). Furthermore, as in the area of other data mining tasks, indications exist of certain reproducibility problems in today's research practice connected with the M&S under consideration Balog et al (2022).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Synthetic Data-Based Simulators for Recommender Systems: A Survey

Stavinova¹,

Grigorievskiy²,

Volodkevich³

et al. 2022

Preprint

View full text Add to dashboard Cite

This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation (M&S) of interactions between users and recommender systems and applications of the M&S to the performance improvement of industrial recommender engines. We start with the motivation behind the development of frameworks implementing the simulations -simulators -and the usage of them for training and testing recommender systems of different types (including Reinforcement Learning ones). Furthermore, we provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness and moreover make a summary of the simulators found in the research literature. Besides other things, we discuss the building blocks of simulators: methods for synthetic data (user, item, user-item responses) generation, methods for what-if experimental analysis, methods and datasets used for simulation quality evaluation (including the methods that monitor and/or close possible simulation-to-reality gaps), and methods for summarization of experimental simulation results. Finally, this survey considers emerging topics and open problems in the field.

show abstract

Section: Simulator's Approbation Reviewmentioning

confidence: 89%

Section: Simulator's Approbationmentioning

confidence: 99%

Section: Simulator's Approbation Reviewmentioning

confidence: 99%

Section: Simulator's Approbation Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Synthetic Data-Based Simulators for Recommender Systems: A Survey

Stavinova¹,

Grigorievskiy²,

Volodkevich³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Influencing recommendation algorithms to reduce the spread of unreliable news by encouraging humans to fact-check articles, in a field experiment

Matias

2023

Sci Rep

View full text Add to dashboard Cite

Society often relies on social algorithms that adapt to human behavior. Yet scientists struggle to generalize the combined behavior of mutually-adapting humans and algorithms. This scientific challenge is a governance problem when algorithms amplify human responses to falsehoods. Could attempts to influence humans have second-order effects on algorithms? Using a large-scale field experiment, I test if influencing readers to fact-check unreliable sources causes news aggregation algorithms to promote or lessen the visibility of those sources. Interventions encouraged readers to fact-check articles or fact-check and provide votes to the algorithm. Across 1104 discussions, these encouragements increased human fact-checking and reduced vote scores on average. The fact-checking condition also caused the algorithm to reduce the promotion of articles over time by as much as −25 rank positions on average, enough to remove an article from the front page. Overall, this study offers a path for the science of human-algorithm behavior by experimentally demonstrating how influencing collective human behavior can also influence algorithm behavior.

show abstract

Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems

Cited by 2 publications

References 11 publications

Synthetic Data-Based Simulators for Recommender Systems: A Survey

Synthetic Data-Based Simulators for Recommender Systems: A Survey

Influencing recommendation algorithms to reduce the spread of unreliable news by encouraging humans to fact-check articles, in a field experiment

Contact Info

Product

Resources

About