A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

Kille, Benjamin; Lommatzsch, Andreas; Hopfgartner, Frank; Larson, Martha; Vries, Arjen P. de

doi:10.1145/3077136.3080726

Cited by 4 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. NewsREEL allows for the evaluation of news recommender algorithms by either "replaying" a news recommendation situation using a static data set, or by providing access to a live data stream of user requests for news articles [Kille et al 2017]. The key component for the online setting that is introduced in Section 3.1.6 is the Open Recommender Platform (ORP) .…”

Section: Open Recommendation Platform (Orp)mentioning

confidence: 99%

Evaluation-as-a-Service for the Computational Sciences

Hopfgartner

Hanbury

Müller

et al. 2018

J. Data and Information Quality

Self Cite

View full text Add to dashboard Cite

Evaluation in empirical computer science is essential to show progress and assess technologies developed. Several research domains such as information retrieval have long relied on systematic evaluation to measure progress: here, the Cranfield paradigm of creating shared test collections, defining search tasks, and collecting ground truth for these tasks has persisted up until now. In recent years, however, several new challenges have emerged that do not fit this paradigm very well: extremely large data sets, confidential data sets as found in the medical domain, and rapidly changing data sets as often encountered in industry. Also, crowdsourcing has changed the way that industry approaches problem-solving with companies now organizing challenges and handing out monetary awards to incentivize people to work on their challenges, particularly in the field of machine learning. This paper is based on discussions at a workshop on Evaluation-as-a-Service (EaaS). EaaS is the paradigm of not providing data sets to participants and have them work on the data locally, but keeping the data central and allowing access via Application Programming Interfaces (API), Virtual Machines (VM) or other possibilities to ship executables. The objectives of this paper are to summarize and compare the current approaches and consolidate the experiences of these approaches to outline the next steps of EaaS, particularly towards sustainable research infrastructures. The paper summarizes several existing approaches to EaaS and analyzes their usage scenarios and also the advantages and disadvantages. The many factors influencing EaaS are overviewed, and the environment in terms of motivations for the various stakeholders, from funding agencies to challenge organizers, researchers and participants, to industry interested in supplying real-world problems for which they require solutions.

show abstract

Section: Open Recommendation Platform (Orp)mentioning

confidence: 99%

Evaluation-as-a-Service for the Computational Sciences

Hopfgartner

Hanbury

Müller

et al. 2018

J. Data and Information Quality

Self Cite

View full text Add to dashboard Cite

show abstract

“…Existing studies performed offline evaluation and online evaluation separately (Gunawardana and Shani, 2009; Shani and Gunawardana, 2011; Garcin and Faltings, 2013; Freyne and Berkovsky, 2013; Gavalas et al , 2014; Hofmann et al , 2016; Kille et al , 2017). Offline evaluation recommends the most popular items, while this strategy is the poorest in an online evaluation; thus, our study is consistent with the literature (Garcin and Faltings, 2013).…”

Section: Discussionmentioning

confidence: 99%

A rule-based automated machine learning approach in the evaluation of recommender engine

Behera

Bala

Jain

2020

BIJ

View full text Add to dashboard Cite

PurposeAny business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain. The purpose of this paper is to choose RE that fits best from a set of candidate solutions using rule-based automated machine learning (ML) approach. The objective is to draw trustworthy conclusion, which results in brand building, and establishing a reliable relation with customers and undeniably to grow the business.Design/methodology/approachAn experimental quantitative research method was conducted in which the ML model was evaluated with diversified performance metrics and five RE algorithms by combining offline evaluation on historical and simulated movie data set, and the online evaluation on business-alike near-real-time data set to uncover the best-fitting RE.FindingsThe rule-based automated evaluation of RE has changed the testing landscape, with the removal of longer duration of manual testing and not being comprehensive. It leads to minimal manual effort with high-quality results and can possibly bring a new revolution in the testing practice to start a service line “Machine Learning Testing as a service” (MLTaaS) and the possibility of integrating with DevOps that can specifically help agile team to ship a fail-safe RE evaluation product targeting SaaS (software as a service) or cloud deployment.Research limitations/implicationsA small data set was considered for A/B phase study and was captured for ten movies from three theaters operating in a single location in India, and simulation phase study was captured for two movies from three theaters operating from the same location in India. The research was limited to Bollywood and Ollywood movies for A/B phase, and Ollywood movies for simulation phase.Practical implicationsThe best-fitting RE facilitates the business to make personalized recommendations, long-term customer loyalty forecasting, predicting the company's future performance, introducing customers to new products/services and shaping customer's future preferences and behaviors.Originality/valueThe proposed rule-based ML approach named “2-stage locking evaluation” is self-learned, automated by design and largely produces time-bound conclusive result and improved decision-making process. It is the first of a kind to examine the business domain and task of interest. In each stage of the evaluation, low-performer REs are excluded which leads to time-optimized and cost-optimized solution. Additionally, the combination of offline and online evaluation methods offer benefits, such as improved quality with self-learning algorithm, faster time to decision-making by significantly reducing manual efforts with end-to-end test coverage, cognitive aiding for early feedback and unattended evaluation and traceability by identifying the missing test metrics coverage.

show abstract

“…Explicit attention to minimal necessary data will promote the development of such algorithms. We note that algorithms that use minimal personal data are useful to address news recommendation, where user IDs might be unstable or unavailable [16].…”

Section: Doing More With Lessmentioning

confidence: 99%

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Larson¹,

Zito²,

Loni³

et al.

Boise State ScholarWorks

Self Cite

View full text Add to dashboard Cite

This paper states the case for the principle of minimal necessary data: If two recommender algorithms achieve the same effectiveness, the better algorithm is the one that requires less user data. Applying this principle involves carrying out training data requirements analysis, which we argue should be adopted as best practice for the development and evaluation of recommender algorithms. We take the position that responsible recommendation is recommendation that serves the people whose data it uses. To minimize the imposition on users' privacy, it is important that a recommender system does not collect or store more user information than it absolutely needs. Further, algorithms using minimal necessary data reduce training time and address the cold start problem. To illustrate the trade-off between training data volume and accuracy, we carry out a set of classic recommender system experiments. We conclude that consistently applying training data requirements analysis would represent a relatively small change in researchers' current practices, but a large step towards more responsible recommender systems. ACM Reference format:

show abstract

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

Cited by 4 publications

References 7 publications

Evaluation-as-a-Service for the Computational Sciences

Evaluation-as-a-Service for the Computational Sciences

A rule-based automated machine learning approach in the evaluation of recommender engine

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Contact Info

Product

Resources

About