Statement frequency coverage: A code coverage criterion for assessing test suite effectiveness

Aghamohammadi, Alireza; Mirian-Hosseinabadi, Seyed-Hassan; Jalali, Sajad

doi:10.1016/j.infsof.2020.106426

Cited by 12 publications

(7 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Aghamohammadi et al [19] have introduced a new code coverage criterion, called statement frequency coverage, to measure test suite effectiveness. They show that this metric outperforms statement and branch coverage.…”

Section: Test Effectiveness Measurementmentioning

confidence: 99%

Learning to predict test effectiveness

Zakeri‐Nasrabadi

Parsa

2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

The high cost of the test can be dramatically reduced, provided that the coverability as an inherent feature of the code under test is predictable. This article offers a machine learning model to predict the extent to which the test could cover a class in terms of a new metric called Coverageability. The prediction model consists of an ensemble of four regression models. The learning samples consist of feature vectors, where features are source code metrics computed for a class. The samples are labeled by the Coverageability values computed for their corresponding classes. We offer a mathematical model to evaluate test effectiveness in terms of size and coverage of the test suite generated automatically for each class. We extend the size of the feature space by introducing a new approach to define submetrics in terms of existing source code metrics. Using feature importance analysis on the learned prediction models, we sort sources code metrics in the order of their impact on the test effectiveness. As a result of which we found the class strict cyclomatic complexity as the most influential source code metric. Our experiments with our prediction models on a large corpus of Java projects containing about 23,000 classes demonstrate the Mean Absolute Error (MAE) of 0.032, Mean-Squared Error (MSE) of 0.004, and an R 2 score of 0.855. Compared with the state-of-the-art coverage prediction models, our models improve MAE, MSE, and an R 2 score by 5.78%, 2.84%, and 20.71%, respectively.

show abstract

Section: Test Effectiveness Measurementmentioning

confidence: 99%

Learning to predict test effectiveness

Zakeri‐Nasrabadi

Parsa

2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

show abstract

“…Code coverage is a metric used in software test analysis to identify the number of lines of source code that have been tested [ 14 ]. The use of code coverage in our study is slightly distinct from how it has been seen in software engineering—when test cases and, in general, unit tests [ 36 ] are used to minimize errors and risks involved with the software development process [ 37 ].…”

Section: Background: Definitions and Conceptsmentioning

confidence: 99%

“…These elements are, for example, lines of code, instructions, blocks of code, functions and procedures and can consider both source code and executable code [ 36 , 65 ]. Throughout this article, we consider the code coverage based on lines of software source code, which is a metric traditionally used to assess the test cases effectiveness [ 14 ].…”

Section: First Round: Software Analysis Interlaboratory Comparison Via Code Coveragementioning

confidence: 99%

“…In the present work, we evaluate the use of two quantitative metrics in order to make the objective comparison between the participating laboratories feasible. We propose metrics based on concepts from SE: code coverage [ 14 ] and software mutation [ 15 ]. The proposed approach was evaluated by executing two interlaboratory comparison rounds with the participation of software analysis and testing accredited laboratories and demonstrated the feasibility of establishing quantitative comparison among those labs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Quantitative Metrics for Performance Monitoring of Software Code Analysis Accredited Testing Laboratories

Chapetta

Neves

Machado

2021

Sensors

View full text Add to dashboard Cite

Modern sensors deployed in most Industry 4.0 applications are intelligent, meaning that they present sophisticated behavior, usually due to embedded software, and network connectivity capabilities. For that reason, the task of calibrating an intelligent sensor currently involves more than measuring physical quantities. As the behavior of modern sensors depends on embedded software, comprehensive assessments of such sensors necessarily demands the analysis of their embedded software. On the other hand, interlaboratory comparisons are comparative analyses of a body of labs involved in such assessments. While interlaboratory comparison is a well-established practice in fields related to physical, chemical and biological sciences, it is a recent challenge for software assessment. Establishing quantitative metrics to compare the performance of software analysis and testing accredited labs is no trivial task. Software is intangible and its requirements accommodate some ambiguity, inconsistency or information loss. Besides, software testing and analysis are highly human-dependent activities. In the present work, we investigate whether performing interlaboratory comparisons for software assessment by using quantitative performance measurement is feasible. The proposal was to evaluate the competence in software code analysis activities of each lab by using two quantitative metrics (code coverage and mutation score). Our results demonstrate the feasibility of establishing quantitative comparisons among software analysis and testing accredited laboratories. One of these rounds was registered as formal proficiency testing in the database—the first registered proficiency testing focused on code analysis.

show abstract

“…Each faulty version of the program is called a mutant. Mutation testing is generally used to assess the effectiveness of a test suite [4][5][6][7][8][9][10][11][12][13][14][15][16][17]. The test suite is executed against generated mutants.…”

Section: Introductionmentioning

confidence: 99%

An ensemble‐based predictive mutation testing approach that considers impact of unreached mutants

Aghamohammadi

Mirian-Hosseinabadi

2021

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Predictive mutation testing (PMT) is a technique to predict whether a mutant is killed, using machine learning approaches. Researchers have proposed various methods for PMT over the years. However, the impact of unreached mutants on PMT is not fully addressed. A mutant is unreached if the statement on which the mutant is generated is not executed by any test cases. We aim at showing that unreached mutants can inflate PMT results. Moreover, we propose an alternative approach to PMT, suggesting a different interpretation for PMT. To this end, we replicated the previous PMT research. We empirically evaluated the suggested approach on 654 Java projects provided by prior literature. Our results indicate that the performance of PMT drastically decreases in terms of area under a receiver operating characteristic curve (AUC) from 0.833 to 0.517. Furthermore, PMT performs worse than random guesses on 27% of the projects. The proposed approach improves the PMT results, achieving the average AUC value of 0.613. As a result, we recommend researchers to remove unreached mutants when reporting the results.

show abstract

Statement frequency coverage: A code coverage criterion for assessing test suite effectiveness

Cited by 12 publications

References 33 publications

Learning to predict test effectiveness

Learning to predict test effectiveness

Quantitative Metrics for Performance Monitoring of Software Code Analysis Accredited Testing Laboratories

An ensemble‐based predictive mutation testing approach that considers impact of unreached mutants

Contact Info

Product

Resources

About