A Systematic Review on Cloud Testing

Abstract

A systematic literature review is presented that surveyed the topic of cloud testing over the period 2012--2017. Cloud testing can refer either to testing cloud-based systems (testing of the cloud) or to leveraging the cloud for testing purposes (testing in the cloud): both approaches (and their combination into testing of the cloud in the cloud) have drawn research interest. An extensive paper search was conducted by both automated query of popular digital libraries and snowballing, which resulted in the final selection of 147 primary studies. Along the survey, a framework has been incrementally derived that classifies cloud testing research among six main areas and their topics. The article includes a detailed analysis of the selected primary studies to identify trends and gaps, as well as an extensive report of the state-of-the-art as it emerges by answering the identified Research Questions. We find that cloud testing is an active research field, although not all topics have received enough attention and conclude by presenting the most relevant open research challenges for each area of the classification framework.

RETORCH: Resource-Aware End-to-End Test Orchestration

“…Moving the testing to the Cloud [4] is commonly acknowledged as a solution to reduce the cost of testing, especially to exploit the potential of unlimited resources and scalability delivered on demand. ElasTest [5] is an open source platform aimed to support Cloud testing and simplify the E2E test process.…”

Cloud testing automation: industrial needs and ElasTest response

“…By leveraging the cloud, testers can use – for the sake of testing – powerful elastic technologies and virtually unlimited resources. Indeed, cloud computing can help reduce the high costs of testing by: (i) not requiring to develop and maintain the testing infrastructure; (ii) exploiting the apparently unlimited computing resources in the cloud while at the same time increasing reusability and automation; (iii) leveraging on‐demand scalable resources for configuration (by using cloud virtualisation) and performance (thanks to cloud elasticity) testing; (iv) emulating various non‐functional properties; (v) providing on‐demand third‐party test services to conduct online validation for large‐scale software systems [3 ].…”

“…Even in cloud testing is a very effective solution for increasing testing performance and reducing its overall effort and cost, it does not come without challenges. Referring to [3 ] for an extensive overview, the main cloud testing challenges can be summarised as: The continuous and rapid evolution of cloud technology forces testing activities to be continuously adapted to the new environment constraints and conditions. The large dimensions and high complexity of cloud systems and their many possible configurations increase the complexity of test design. Indeed cloud environment features such as heterogeneity, scalability, load balancing, communication, and synchronisation need to be taken into consideration during test design. Contrariwise to expectations, the adoption of wrong strategies in the allocation of the virtual machines or the loads balancing could result in an increase of testing time.…”

“…The already hard oracle problem could become more challenging, as effective oracle solutions, as well as self‐reseTable and auto‐recoverable test scripts, are needed to support and process the test failures during automated test execution. Additionally, improved features linking the cloud test reports with information/metrics that could help testers in fault identification and removal are still a challenging aspect. Despite the listed challenges and open issues, as extensively surveyed in [3 ], the usage of cloud infrastructures for testing has several advantages. We summarise here the main ones that are: Cost reduction: Cloud resources can be exploited to achieve test cost saving, scalability, and efficient utilisation of the test resources while guaranteeing a high quality of service. Improving resources utilisation: The usage of virtual machines allow for optimising the cloud resources and saving configuration effort. Speeding up test execution: Cloud infrastructure supports the concurrent execution of test cases by either partitioning them into tasks or distributing them into different cloud processors.…”

Shared-Mode Resource Allocation for Cloud-Based Load Testing

“…Nowadays, load testing is often migrated to the cloud and conducted based on cloud resources. Performing load testing in the cloud can ease the setup of test environments and reduce the hardware purchase and maintenance cost [5]- [7].…”