QoS-based ranking and selection of SaaS applications using heterogeneous similarity metrics

Abstract

The plethora of cloud application services (Apps) in the cloud business apps e-marketplace often leads to service choice overload. Meanwhile, existing SaaS e-marketplaces employ keyword-based inputs that do not consider both the quantitative and qualitative quality of service (QoS) attributes that characterise cloud-based services. Also, existing QoS-based cloud service ranking approaches rank cloud application services are based on the assumption that the services are characterised by quantitative QoS attributes alone, and have employed quantitative-based similarity metrics for ranking. However, the dimensions of cloud service QoS requirements are heterogeneous in nature, comprising both quantitative and qualitative QoS attributes, hence a cloud service ranking approach that embrace core heterogeneous QoS dimensions is essential in order to engender more objective cloud selection. In this paper, we propose the use of heterogeneous similarity metrics (HSM) that combines quantitative and qualitative dimensions for QoS-based ranking of cloud-based services. By using a synthetically generated cloud services dataset, we evaluated the ranking performance of five HSM using Kendall tau rank coefficient and precision as accuracy metrics benchmarked with one HSM. The results show significant rank order correlation of Heterogeneous Euclidean-Eskin Metric, Heterogeneous Euclidean-Overlap Metric, and Heterogeneous Value Difference Metric with human similarity judgment, compared to other metrics used in the study. Our results confirm the applicability of HSM for QoS ranking of cloud services in cloud service e-marketplace with respect to users' heterogeneous QoS requirements.

A Taxonomy of Quality Metrics for Cloud Services

“…It can be measured with continuous values (between 0 and 1). Qualitative evaluations are those that indicate qualities or qualitative categories (e.g., an attribute such as the flexibility of a cloud service that rates the ability to add or remove predefined features from service in order to customize it [34]. It can be measured as High, Medium, or Low).…”

“…These metrics can be applied to cloud artifacts obtained in different lifecycle phases (e.g., design, document, test cases) that implement a particular change. Examples of these metrics include flexible force, which measures the ease or difficulty with which a service can be changed as a response to a customer request [47], and rating the ability to add or remove predefined features from a service in order to accommodate users' preferences [34].…”

“…However, our main concern is that usability was generally measured subjectively by employing qualitative measures. For example, Ezenwoke et al [34] measured it as the ease with which a cloud service can be used, learned, operated, installed, and understood by the user. Moreover, the current measurement efforts are not focused on users; the metrics are, in most cases, subjective and based on estimates of past user experiences.…”

Benchmark Dataset Selection of Web Services Technologies: A Factor Analysis

“…The dataset D [33] is a simulated dataset of web services that contains a total of six quality metrics. Dataset D is a mixture of qualitative and quantitative quality metrics.…”

“…Response Time along with other quality metrics, is mentioned as a quality metric in the remaining two datasets. Dataset meta information in the form of WSDL is available for datasets A, B, and C, while it is not mentioned for the dataset D. It is because the authors [33] generated dataset D for the evaluation of cloud web services. Another difference between the four datasets is that only dataset A mentioned the geographic location of web services and users.…”