Requirements based test prioritization using risk factors: An industrial study

Abstract

Context: Software testing is an expensive and time-consuming process. Software engineering teams are often forced to terminate their testing efforts due to budgetary and time constraints, which inevitably lead to long term issues with quality and customer satisfaction. Test case prioritization (TCP) has shown to improve test effectiveness. Objective: The results of our prior work on requirements-based test prioritization showed improved rate of fault detection on industrial projects; the customer priority (CP) and the fault proneness (FP) were the biggest contributing factors to test effectiveness. The objective of this paper is to further investigate these two factors and apply prioritization based on these factors in a different domain: an enterprise level cloud application. We aim to provide an effective prioritization scheme that practitioners can implement with minimum effort. The other objective is to compare the results and the benefits of these two factors with two risk-based prioritization approaches that extract risks from the system requirements categories. Method: Our approach involved analyzing and assigning values to each requirement based on two important factors, CP and FP, so that the test cases for high-value requirements are prioritized earlier for execution. We also proposed two requirements-based TCP approaches that use risk information of the system. Results: Our results indicate that the use of CP and FP can improve the effectiveness of TCP. The results also show that the risk-based prioritization can be effective in improving the TCP. Conclusion: We performed an experiment on an enterprise cloud application to measure the fault detection rate of different test suites that are prioritized based on CP, FP, and risks. The results depict that all approaches outperform the random prioritization approach, which is prevalent in the industry. Furthermore, the proposed approaches can easily be used in the industry to address the schedule and budget constraints at the testing phase.

Automating system test case classification and prioritization for use case-driven testing in product lines

“…Test case prioritization techniques schedule test cases in an order that increases their effectiveness in meeting some performance goals (e.g., rate of fault detection and number of test cases required to discover all the faults) [51,91,115]. They mostly use information about previous executions of test cases (e.g., [29,33,44,60,63,91]), human knowledge (e.g., [8,54,96,97,99,103]), or a model of the system under test (e.g., [34,53,55,101]). For instance, Shrikanth et al [98] propose a test case prioritization approach that takes into consideration customer-assigned priorities of requirements, developer-perceived implementation complexity, requirements volatility, and fault proneness of requirements.…”

Novel Fuzzy Clustering Methods for Test Case Prioritization in Software Projects

“…Thomas et al [23] proposed text analysis method for static TCP. In this paper, the main objective of the similarity-based techniques is to maximize the commonality and minimize the diversity, which is the opposite to researchers that have been obtaining priority that is based on maximizing the diversity and minimizing the similarity [24][25][26][27]. In general, similarity-based TCP is mainly divided into two types, which are distribution based and ART (Adaptive Random Test) inspired.…”

Value-Driven V-Model: From Requirements Analysis to Acceptance Testing

“…Yoon et al [42] proposed a method that prioritizes test cases through a correlation of requirement and risk. Srikanth et al [43] proposed requirements-based test prioritization with customer severity, fault proneness, requirements volatility, and implementation complexity as risk factors. Li et al [20] proposed a value-based testing method that prioritizes testing based on business importance, quality risk, and testing cost.…”