Generic Methods for Multi-criteria Evaluation

Lavesson, Niklas; Davidsson, Paul

doi:10.1137/1.9781611972788.49

Cited by 9 publications

(10 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to the ACC and AUC metrics we will evaluate each algorithm using the Candidate Evaluation Function (CEF) [23]. The purpose of CEF is to capture application-specific tradeoffs by combining multiple relevant metrics.…”

Section: Multi-criteria Evaluationmentioning

confidence: 99%

Learning to detect spyware using end user license agreements

et al. 2010

Self Cite

View full text Add to dashboard Cite

The amount of software that hosts spyware has increased dramatically. To avoid legal repercussions, the vendors need to inform users about inclusion of spyware via end user license agreements (EULAs) during the installation of an application. However, this information is intentionally written in a way that is hard for users to comprehend. We investigate how to automatically discriminate between legitimate software and spyware associated software by mining EULAs. For this purpose, we compile a data set consisting of 996 EULAs out of which 9.6% are associated to spyware. We compare the performance of 17 learning algorithms with that of a baseline algorithm on two data sets based on a bag-of-words and a meta data model. The majority of learning algorithms significantly outperform the baseline regardless of which data representation is used. However, a non-parametric test indicates that bag-of-words is more suitable than the meta model. Our conclusion is that automatic EULA classification can be applied to assist users in making informed decisions about whether to install an application without having read the EULA. We therefore outline the design of a spyware prevention tool and suggest how to select suitable learning algorithms for the tool by using a multi-criteria evaluation approach.

show abstract

Section: Multi-criteria Evaluationmentioning

confidence: 99%

Learning to detect spyware using end user license agreements

et al. 2010

Self Cite

View full text Add to dashboard Cite

show abstract

“…In previous work [7], we identified a number of attractive properties of existing multi-criteria evaluation metrics and presented a generic multi-criteria metric that was designed with these properties in mind. This metric, called the Candidate Evaluation Function (CEF), has the main purpose of combining an arbitrary number of individual metrics into a single quantity.…”

Section: Application-oriented Validation and Evaluationmentioning

confidence: 99%

“…As discussed in [7], one study proposes the SAR metric, which combines squared error, success rate, and AUC. This metric may be suitable for a certain application, however, it cannot be used to compare other criteria.…”

Section: Introductionmentioning

confidence: 99%

APPrOVE: Application-oriented validation and evaluation of supervised learners

Lavesson

Davidsson

2010

2010 5th IEEE International Conference Intelligent Systems

Self Cite

View full text Add to dashboard Cite

“…For this purpose we suggest the generic multi-criteria metric, CEF [18], which can be used to trade-off multiple evaluation metrics when evaluating or selecting between different learning algorithms or classifiers. Each included metric can be associated with an explicit weight and an acceptable range.…”

Section: Multi-criteria Evaluationmentioning

confidence: 99%

“…When analyzing the prevention tool requirements it is clear that we need to use evaluation metrics for accuracy (of classifying both good and bad applications), time (classification response time), and explainability (for visualization). Mapping the requirements to the available experiment data and making informed choices (see [18]) about bounds and explicit weighting, we can calculate the CEF score for all algorithms included in the experiment. This score is presented in the last column of Tab.…”

Section: Multi-criteria Evaluationmentioning

confidence: 99%

Spyware Prevention by Classifying End User License Agreements

Lavesson

Davidsson

Boldt

et al.

New Challenges in Applied Intelligence Technologies

Self Cite

View full text Add to dashboard Cite

Abstract. We investigate the hypothesis that it is possible to detect from the End User License Agreement (EULA) if the associated software hosts spyware. We apply 15 learning algorithms on a data set consisting of 100 applications with classified EULAs. The results show that 13 algorithms are significantly more accurate than random guessing. Thus, we conclude that the hypothesis can be accepted. Based on the results, we present a novel tool that can be used to prevent spyware by automatically halting application installers and classifying the EULA, giving users the opportunity to make an informed choice about whether to continue with the installation. We discuss positive and negative aspects of this prevention approach and suggest a method for evaluating candidate algorithms for a future implementation.

show abstract

Generic Methods for Multi-criteria Evaluation

Cited by 9 publications

References 5 publications

Learning to detect spyware using end user license agreements

Learning to detect spyware using end user license agreements

APPrOVE: Application-oriented validation and evaluation of supervised learners

Spyware Prevention by Classifying End User License Agreements

Contact Info

Product

Resources

About