Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software

The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI Builds

“…While CI is advantageous for projects, other studies revealed the difficulties of adopting CI. Among the several challenges that come with CI, build failures are a challenge that is unavoidable for many projects [60,59,83,45,52,84,76,25]. For example, Kerzazi et al [45] analyzed 3,214 builds produced in a large software company over a period of 6 months.…”

Feature-oriented Test Case Selection during Evolution of Highly-Configurable Systems

“…Prado Lima et al [8,50] introduce two strategies to apply a TCP learning-based approach called COLEMAN in the CI of HCSs: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant; and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. COLEMAN is an approach that learns from the test case failure-history, guided by a reward function.…”

“…In summary, existing work present the following main limitations: (i) are dependent on models or other artifacts (e.g., FM or HCS architecture) that may not exist or be outdated [14,15,16,17,18,39,40,41]; (ii) require a failure-history or dynamic analysis [22,23,24,50,8], what is costly and may be not suitable for a CI scenario; (iii) do not consider HCS particularities and/or languages such as C and C++, largely adopted for the HCS development [26,27,28,49]; (iv) consider that there is a kind of mapping from code to the test cases or that the HCSs are developed following a specific format [22,25]; and (v) do not work with the concept of features, which are fundamental units of design and communication in the HCS context [46,47,48,29]. Our approach, presented in Section 4, addresses these limitations.…”

An Improved Method for Test Case Prioritization in Continuous Integration based on Reinforcement Learning

“…There has been a lot of research on test case prioritization in continuous integration environment. Lima et al [16] improves COLEMAN, a learning-based sorting method, and puts forward two strategies to deal with variables, which makes COLEMAN practicable for highly-configurable software in continuous integration; Rosenbauer et al [17] proposed and optimized a test case ranking model based on Learning Classification System (LCS), and demonstrated through experiments that it performs better than network-based models; Xiao et al [18] proposed a sorting method based on Long Short-Term Memory Network (LSTM) and applied it to embedded software, which improved its fault detection rate in the continuous integration environment; Ali et al [19] proposed a clustering method that clusters and sorts test cases based on their historical failure frequency and coverage criteria, achieving a defect detection rate of over 90%.…”