Vertical Test Reuse for Embedded Systems: A Systematic Mapping Study

Flemström, Daniel; Sundmark, Daniel; Afzal, Wasif

doi:10.1109/seaa.2015.46

Cited by 8 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Systematic mapping studies are recommended methods for getting a broad understanding of a research topic and does not involve detailed synthesis as in the case of a systematic literature review (see e.g. [25], [26]). Our methodology is driven by using a predefined protocol that aims to be unbiased by being auditable and repeatable [27].…”

Section: Research Methodology a Systematic Mapping Studymentioning

confidence: 99%

On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering

et al. 2020

Self Cite

View full text Add to dashboard Cite

The inclusion of grey literature (GL) is important to remove publication bias while gathering available evidence regarding a certain topic. The number of systematic literature reviews (SLRs) in Software Engineering (SE) is increasing but we do not know about the extent of GL usage in these SLRs. Moreover, Google Scholar is rapidly becoming a search engine of choice for many researchers but the extent to which it can find the primary studies is not known. Objective: This tertiary study is an attempt to i) measure the usage of GL in SLRs in SE. Furthermore this study proposes strategies for categorizing GL and a quality checklist to use for GL in future SLRs; ii) explore if it is feasible to use only Google Scholar for finding scholarly articles for academic research. Method: We have conducted a systematic mapping study to measure the extent of GL usage in SE SLRs as well as to measure the feasibility of finding primary studies using Google Scholar. Results and conclusions: a) Grey Literature: 76.09% SLRs (105 out of 138) in SE have included one or more GL studies as primary studies. Among total primary studies across all SLRs (6307), 582 are classified as GL, making the frequency of GL citing as 9.23%. The intensity of GL use indicate that each SLR contains 5 primary studies on average (total intensity of GL use being 5.54). The ranking of GL tells us that conference papers are the most used form 43.3% followed by technical reports 28.52%. Universities, research institutes, labs and scientific societies together make up 67.7% of GL used, indicating that these are useful sources for searching GL. We additionally propose strategies for categorizing GL and criteria for evaluating GL quality, which can become a basis for more detailed guidelines for including GL in future SLRs. b) Google Scholar Results: The results show that Google Scholar was able to retrieve 96% of primary studies of these SLRs. Most of the primary studies that were not found using Google Scholar were from grey sources.

show abstract

Section: Research Methodology a Systematic Mapping Studymentioning

confidence: 99%

On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…The work from Flemström [20] deals with a very specific research area: vertical test reuse for embedded systems. They defined vertical reuse as the reuse of artifacts over different integration levels during the engineering process.…”

Section: Related Workmentioning

confidence: 99%

What software reuse benefits have been transferred to the industry? A systematic mapping study

Barros-Justo

Pinciroli

Matalonga

et al. 2018

Information and Software Technology

View full text Add to dashboard Cite

Context: The term software reuse was first used in 1968 at the NATO conference. Since then, work in the scientific literature have stated that the application of software reuse offers benefits such as increase in quality and productivity. Nonetheless, in spite of many publications reporting software reuse experiences, evidence that such benefits having reached industrial settings is scarce. Objective: To identify and classify the benefits transferred to real-world settings by the application of software reuse strategies. Method: We conducted a systematic mapping study (SMS). Our search strategies retrieved a set of 2,413 papers out of which 49 were selected as primary studies. We defined five facets to classify these studies: a) the type of benefit, b) the reuse process, c) the industry's domain, d) the type of reuse and e) the type of research reported. Results: Quality increase (28 papers) and Productivity increase (25 papers) were the two most mentioned benefits. Component-Based Development (CBD) was the most reported reuse strategy (41%), followed by Software Product Lines (SPL, 30%). The selected papers mentioned fourteen industrial domains, of which four stand out: aerospace and defense, telecommunications, electronics and IT services. The application of systematic reuse was reported in 78% of the papers. Regarding the research type, 50% use evaluation research as the investigation method. Finally, 13 papers (27%) reported validity threats for the research method applied. Conclusions: The literature analyzed presents a lack of empirical data, making it difficult to evaluate the effective transfer of benefits to the industry. This work did not find any relationship between the reported benefits and the reuse strategy applied by the industry or the industry domain. Although the most reported research method was industrial case studies (25 works), half of these works (12) did not report threats to validity.

show abstract

“…Even though similarities, both within and among test cases, can be identified, our previous study (Flemström et al 2015) showed that these overlaps can be scattered all over the set of test cases and could not be trivially removed. Further, the study indicated that test cases seldom overlap entirely.…”

Section: Introductionmentioning

confidence: 92%

Similarity-based prioritization of test case automation

et al. 2018

Self Cite

View full text Add to dashboard Cite

The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches Software Qual J at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.

show abstract

Vertical Test Reuse for Embedded Systems: A Systematic Mapping Study

Cited by 8 publications

References 26 publications

On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering

On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering

What software reuse benefits have been transferred to the industry? A systematic mapping study

Similarity-based prioritization of test case automation

Contact Info

Product

Resources

About