Dimensions and Metrics for Evaluating Recommendation Systems

Avazpour, Iman; Pitakrat, Teerat; Grunske, Lars; Grundy, John

doi:10.1007/978-3-642-45135-5_10

Cited by 59 publications

(50 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, before studies evaluate the interplay between users and tools, it is unclear if specialized solutions are worth the additional development eort required. This is in line with discussions on improved tool support for trace recovery (Borg and Pfahl, 2011), and the dierence of correctness and utility of recommendation systems in software engineering (Avazpour et al, 2014).…”

Section: Other Approaches To Automated Bug Assignmentsupporting

confidence: 75%

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

et al. 2015

View full text Add to dashboard Cite

Context: Bug report assignment is an important part of software maintenance. In particular, incorrect assignments of bug reports to development teams can be very expensive in large software development projects.Several studies propose automating bug assignment techniques using machine learning in open source software contexts, but no study exists for large-scale proprietary projects in industry. Objective: The goal of this study is to evaluate automated bug assignment techniques that are based on machine learning classication. In particular, we study the state-of-the-art ensemble learner Stacked Generalization (SG) that combines several classiers. Method: We collect more than 50,000 bug reports from ve development projects from two companies in dierent domains. We implement automated bug assignment and evaluate the performance in a set of controlled experiments. Results: We show that SG scales to large scale industrial application and that it outperforms the use of individual classiers for bug assignment, reaching prediction accuracies from 50% to 90% when large training sets are used. In addition, we show how old training data can decrease the prediction accuracy of bug assignment. Conclusions: We advice industry to use SG for bug assignment in proprietary contexts, using at least 2,000 bug reports for training. Finally, we highlight the importance of not solely relying on results from cross-validation when evaluating automated bug assignment.

show abstract

Section: Other Approaches To Automated Bug Assignmentsupporting

confidence: 75%

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Netflix selected a random subset of users from their entire customer base 6] with at least 20 ratings in a given period. A Hold-Out set was created from the 9 most recent ratings of each of these users, 3 consisting of about 4.2 million ratings. The remaining data formed the Training set.…”

Section: Benchmarking and Evaluation Settingsmentioning

confidence: 99%

“…We overview a few such metrics here; for a more complete overview of recommendation evaluation measures, see Chap. 10 [3].…”

Section: Accuracy and Error Metricsmentioning

confidence: 99%

Recommendation Systems in Software Engineering

2014

113

View full text Add to dashboard Cite

This chapter describes the concepts involved in the process of benchmarking of recommendation systems. Benchmarking of recommendation systems is used to ensure the quality of a research system or production system in comparison to other systems, whether algorithmically, infrastructurally, or according to any sought-after quality. Specifically, the chapter presents evaluation of recommendation systems according to recommendation accuracy, technical constraints, and business values in the context of a multi-dimensional benchmarking and evaluation model encompassing any number of qualities into a final comparable metric. The focus is put on quality measures related to recommendation accuracy, technical factors, and business values. The chapter first introduces concepts related to evaluation and benchmarking of recommendation systems, continues with an overview of the current state of the art, then presents the multi-dimensional approach in detail. The chapter concludes with a brief discussion of the introduced concepts and a summary. A. Said ( )

show abstract

“…The major goal of this book chapter is to shed light on the basic properties of the three major recommendation approaches of (1) collaborative filtering [12,26,36], (2) content-based filtering [49], and (3) knowledge-based recommendation [5,16]. Starting with the basic algorithmic approaches, we exemplify the functioning of the algorithms and discuss criteria that help to decide which algorithm should be applied in which context.…”

Section: Introductionmentioning

confidence: 99%

Basic Approaches in Recommendation Systems

Felfernig

Jeran

Ninaus

et al. 2013

Recommendation Systems in Software Engineering

View full text Add to dashboard Cite

Recommendation systems support users in finding items of interest. In this chapter, we introduce the basic approaches of collaborative filtering, contentbased filtering, and knowledge-based recommendation. We first discuss principles of the underlying algorithms based on a running example. Thereafter, we provide an overview of hybrid recommendation approaches which combine basic variants. We conclude this chapter with a discussion of newer algorithmic trends, especially critiquing-based and group recommendation. IntroductionRecommendation systems [7,33] provide suggestions for items that are of potential interest for a user. These systems are applied for answering questions such as which book to buy? [39], which website to visit next? [49], and which financial service to choose? [19]. In software engineering scenarios, typical questions that can be answered with the support of recommendation systems are, for example, which software changes probably introduce a bug? [3], which requirements to implement in the next software release? [25] The major goal of this book chapter is to shed light on the basic properties of the three major recommendation approaches of (1) collaborative filtering [12,26,36], (2) content-based filtering [49], and (3) knowledge-based recommendation [5,16]. Starting with the basic algorithmic approaches, we exemplify the functioning of the algorithms and discuss criteria that help to decide which algorithm should be applied in which context.The remainder of this chapter is organized as follows. In Sect. 2.2 we give an overview of collaborative filtering recommendation approaches. In Sect. 2.3 we introduce the basic concepts of content-based filtering. We close our discussion of basic recommendation approaches with the topic of knowledge-based recommendation (see Sect. 2.4). In Sect. 2.5, we explain example scenarios for integrating the basic recommendation algorithms into hybrid ones. Hints for practitioners interested in the development of recommender applications are given in Sect. 2.6. A short overview of further algorithmic approaches is presented in Sect. 2.7. Collaborative FilteringThe item-set in our running examples is software engineering-related learning material offered, for example, on an e-learning platform (see Table 2.1). Each learning unit is additionally assigned to a set of categories, for example, the learning unit l 1 is characterized by Java and UML.Collaborative filtering [12,36,56] is based on the idea of word-of-mouth promotion: the opinion of family members and friends plays a major role in personal decision making. In online scenarios (e.g., online purchasing [39]), family members and friends are replaced by the so-called nearest neighbors (NN) who are users with a similar preference pattern or purchasing behavior compared to the current user. Collaborative filtering (see Fig. 2.1) relies on two different types of background data: (1) a set of users and (2) a set of items. The relationship between users and items is primarily expressed in terms of ratings wh...

show abstract

Dimensions and Metrics for Evaluating Recommendation Systems

Cited by 59 publications

References 67 publications

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

Recommendation Systems in Software Engineering

Basic Approaches in Recommendation Systems

Contact Info

Product

Resources

About