A Systematic Approach to Evaluating Open Source Software

Abstract

Selecting appropriate Open Source Software (OSS) for a given problem or a set of requirements can be very challenging. Some of the difficulties are due to the fact that there is not a generally accepted set of criteria to use in evaluation and that there are usually many OSS projects available to solve a particular problem. In this study, the authors propose a set of criteria and a methodology for assessing candidate OSS for fitness of purpose using both functional and non-functional factors. The authors then use these criteria in an improved solution to the decision problem using the well-developed Analytical Hierarchy Process. In order to validate the proposed model, it is applied at a technology management company in the United Arab Emirates, which integrates many OSS solutions into its Information Technology infrastructure. The contribution of this work is to help decision makers to better identify an appropriate OSS solution using a systematic approach without the need for intensive performance testing.

FOSSES: Framework for open‐source software evaluation and selection

“…The strength of this approach is that it introduces a feature‐based approach in which the features of given OSS are used as evaluation criteria for deciding among the alternatives. This is in line with how evaluation is done in practice . However, the study adopts the AHP, which is saddled with a number of known flaws .…”

“…The metrics in the decision hierarchy of the OSS category considered were sourced from relevant literature works online. This is one of the approaches suggested in similar studies …”

“…The study by Ahmad and Laplante proposed a set of criteria (consisting of both functional and nonfunctional factors), which were structured into a hierarchical model. The AHP was then used to evaluate and select the most suitable OSS.…”

“…First, the set of criteria chosen is based on the literature, whereas in practice, evaluators adopt an ad hoc approach, such as vendor recommendations, recommendations and review from people in the developers' community, and the engagement of Web search engines. Second, the validation of the approach employed only one scenario, whereas multiple scenarios were desirable as noted by the authors …”

“…A large number of well‐known MCDM methods exist, such as the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), data envelopment analysis, the analytic hierarchy process (AHP), the analytic network process, VIKOR, decision‐making trial and evaluation laboratory, the Preference Ranking Organization Method for Enrichment Evaluations, and the Elimination and Choice Expressing Reality method, to mention a few . The AHP is the most popular in terms of its application in software evaluation and selection . However, the AHP has some known limitations, which can easily be surmounted by using the evidential reasoning (ER) approach in place of the AHP .…”