Cluster-based test scheduling strategies using semantic relationships between test specifications

Abstract

One of the challenging issues in improving the test efficiency is that of achieving a balance between testing goals and testing resources. Test execution scheduling is one way of saving time and budget, where a set of test cases are grouped and tested at the same time. To have an optimal test execution schedule, all related information of a test case (e.g. execution time, functionality to be tested, dependency and similarity with other test cases) need to be analyzed. Test scheduling problem becomes more complicated at high-level testing, such as integration testing and especially in manual testing procedure. Test specifications are generally written in natural text by humans and usually contain ambiguity and uncertainty. Therefore, analyzing a test specification demands a strong learning algorithm. In this position paper, we propose a natural language processing-based approach that, given test specifications at the integration level, allows automatic detection of test cases semantic dependencies. The proposed approach utilizes the Doc2Vec algorithm and converts each test case into a vector in n-dimensional space. These vectors are then grouped using the HDBSCAN clustering algorithm into semantic clusters. Finally, a set of cluster-based test scheduling strategies are proposed for execution. The proposed approach has been applied in a subsystem from the railway domain by analyzing an ongoing testing project at Bombardier Transportation AB, Sweden.

sOrTES: A Supportive Tool for Stochastic Scheduling of Manual Integration Test Cases

“…Similarly, Haidry and Miller [22] prioritized functionally-dependent test cases based on different forms of the graph coverage values. In order to detect functional dependencies among integration test cases at an early stage, our earlier work [23] proposed, using natural language processing (NLP), to analyze multiple related artefacts (test specification, software requirement specification and relevant signaling information between functions under test).…”

“…Analyzing a wide range of test specifications which are written by human with a variance in language and testing skills makes the problem more complex. The problem of dependency detection between manual integration test cases has previously been solved through proposing several approaches such as a questionnaire based study, deep learning, natural language processing (NLP) and machine learning [6], [23], [32]. We also proposed an aiding tool called ESPRET 1 for estimating the execution time for manual integration test cases before the first execution [33].…”

Visualizing Test Diversity to Support Test Optimisation

“…In practice, these two approaches should not significantly change the resulting diversity [5], and would only affect which tests are included first in the subset. Alternatively, other approaches use the information to cluster and then select tests [15,7]. The result of Step 3 is a diverse subset of tests T ∈ T .…”

“…In parallel, several studies investigate the application of diversity-based in different domains, such as model-based testing [8,12,13], continuous integration pipelines [14,2], search-based test generation [18] and at higher levels of testing such as acceptance [23], system [15] and integration [14]. Alternatively, studies also focus on investigating the tradeoff when using different distance functions in distinct sources of diversity such as use cases [5], or modified artefacts [13].…”

“…A variety of techniques support diversity-based testing across different domains [2,12,8,13,3] and levels of testing (unit [1,3], integration [14] and system [15,13]), and many studies investigate their benefits and drawbacks when used for automated test optimisation such as test case prioritisation or selection. On one hand, these techniques show effective fault detection rate for selective testing [8,12,4], while on the other hand, their application can be prohibitive due to costly calculations when used against large sets of test cases or for repeated selection selection [11,7].…”