Exploring Test Overlap in System Integration: An Industrial Case Study

Flemström, Daniel; Afzal, Wasif; Sundmark, Daniel

doi:10.1109/seaa.2016.34

Cited by 5 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related Work: Several studies have been conducted on redundancy and gaps in testing efforts. The large volume of test cases needed for complex systems carries risks of redundancies [10]- [12]. However, redundancy can improve quality, and should not always be avoided [11].…”

Section: Background and Related Workmentioning

confidence: 99%

Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software

Bisht,

Ejigu,

Gay

et al. 2023

2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

Testing of automotive systems usually follows the V-Model, a process where sequential testing activities progress from low-level code structures to high-level integrated systems. In theory, the V-Model should reduce redundant testing and prevent gaps in verification. To assess whether such benefits translate in practice, in a case study at Scania CV AB, we have developed a framework to identify redundancies and gaps in test cases across V-model test levels.Our framework identified both redundancies and gaps in Scania's scripted testing efforts. Deviating cases were also identified where, e.g., requirements were outdated or contained incorrect details. Factors contributing to redundancy include re-verification in a new context, difficulties mapping requirements across levels, and lack of test case documentation. Both redundancies and gaps result from a lack of communication and traceability of test results across test levels. We recommend active collaboration across levels, as well as use of coverage matrices to alleviate these issues. We offer our framework to help refine testing practices and to inspire process improvements.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software

Bisht,

Ejigu,

Gay

et al. 2023

2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

show abstract

“…Many of these studies also propose visualization methods. Some other examples of visualizations in the testing context are given in [8,9] to identify similarities across test cases. In [18], the authors propose an interactive visualization technique for analysis of results of large-scale software regression testing.…”

Section: Background and Related Workmentioning

confidence: 99%

Decision making and visualizations based on test results

Strandberg¹,

Afzal

Sundmark

2018

Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Self Cite

View full text Add to dashboard Cite

Background: Testing is one of the main methods for quality assurance in the development of embedded software, as well as in software engineering in general. Consequently, test results (and how they are reported and visualized) may substantially in uence business decisions in software-intensive organizations. Aims: This case study examines the role of test results from automated nightly software testing and the visualizations for decision making they enable at an embedded systems company in Sweden. In particular, we want to identify the use of the visualizations for supporting decisions from three aspects: in daily work, at feature branch merge, and at release time. Method: We conducted an embedded case study with multiple units of analysis by conducting interviews, questionnaires, using archival data and participant observations. Results: Several visualizations and reports built on top of the test results database are utilized in supporting daily work, merging a feature branch to the master and at release time. Some important visualizations are: lists of failing test cases, easy access to log les, and heatmap trend plots. The industrial practitioners perceived the visualizations and reporting as valuable, however they also mentioned several areas of improvement such as better ways of visualizing test coverage in a functional area as well as better navigation between di erent views. Conclusions: We conclude that visualizations of test results are a vital decision making tool for a variety of roles and tasks in embedded software development, however the visualizations need to be continuously improved to keep their value for its stakeholders. CCS CONCEPTS • Software and its engineering → Software creation and management; Software testing and debugging; Empirical software validation;

show abstract

“…Looking at the stimuli column, we can see that they indeed express the same thing, although slightly reworded. Although we use the technique originally described in our previous work (Flemström et al 2016) to identify such similar test steps, any similarity function that returns the same numerical value for each two similar test steps can be used. For this paper, we have only considered the stimulus of the test steps.…”

Section: Preliminariesmentioning

confidence: 99%

“…The data from the DOORS data base was extracted into a sandbox environment, similar to Flemström et al (2016). After extraction, the major preparation steps included feature extraction, removal of stop words as well as formatting characters and lastly encoding into word vectors.…”

Section: Execution and Data Collection Proceduresmentioning

confidence: 99%

“…At point (c) in time, the automation work is completed and no more manual effort is required for a system test. Given that we can efficiently reuse already automated parts of the test cases, using a technique such as the one suggested in Flemström et al (2016) to identify reusable parts, the automation work could be finished earlier. In Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Similarity-based prioritization of test case automation

et al. 2018

Self Cite

View full text Add to dashboard Cite

The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches Software Qual J at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.

show abstract

Exploring Test Overlap in System Integration: An Industrial Case Study

Cited by 5 publications

References 15 publications

Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software

Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software

Decision making and visualizations based on test results

Similarity-based prioritization of test case automation

Contact Info

Product

Resources

About