How Good Are My Tests?

Bowes, David; Hall, Tracy; Petrić, Jean; Shippey, Thomas; Turhan, Burak

doi:10.1109/wetsom.2017.2

Cited by 32 publications

(38 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hard to comprehend BDD specifications Investigate BDD test smells, technical debt, and the adoption of test suite comprehension techniques to BDD specifications [19], [20], [21], [22], [23], [24], [25] Non-adherence to the BDD workflow Investigate the incorporation of maintenance concerns at the core of BDD workflow and tools [38], [39] O10 Scarcity of coaching and material guidelines on BDD Investigate the impact of coaching and guidelines on producing maintainable specifications [4], [21] TABLE III CHALLENGES AND RESEARCH OPPORTUNITIES FOR MAINTENANCE AND EVOLUTION OF BDD SPECIFICATIONS on implementing correct software behaviours that led to the birth of BDD [1]. However, a comparative study would shed more light on whether that foreseen BDD's potential has become practically evident.…”

Section: O1mentioning

confidence: 99%

“…Also, specifically on duplication, practitioners admitted that the presence of duplication in BDD specification causes, among others, test execution, maintenance and evolution problems. To be of desirable quality, like the code under test, test suites must be maintainable [21], [24], [46], [19], [47]. As such, we argue that it is important to investigate test maintainability aspects such as ease of change, bug detectability (for the bugs in both test and production code) as well as test comprehensibility [24], [21], and others, in the context of BDD, to support the work of practitioners.…”

Section: O1mentioning

confidence: 99%

See 1 more Smart Citation

Maintaining behaviour driven development specifications: Challenges and opportunities

Binamungu

Embury

Κωνσταντίνου

2018

2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

Abstract-In Behaviour-Driven Development (BDD) the behaviour of a software system is specified as a set of example interactions with the system using a "Given-WhenThen" structure. These examples are expressed in high level domain-specific terms, and are executable. They thus act both as a specification of the requirements and as tests that can verify whether the current system implementation provides the desired behaviour or not. This approach has many advantages but also presents some problems. When the number of examples grows, BDD specifications can become costly to maintain and extend. Some teams find that parts of the system are effectively frozen due to the challenges of finding and modifying the examples associated with them. We surveyed 75 BDD practitioners from 26 countries to understand the extent of BDD use, its benefits and challenges, and specifically the challenges of maintaining BDD specifications in practice. We found that BDD is in active use in industry, and that the use of domain specific terms, improving communication among stakeholders, the executable nature of BDD specifications, and facilitating comprehension of code intentions are the main benefits of BDD. The results also showed that BDD specifications suffer the same maintenance challenges found in automated test suites generally. We mapped the survey results to the literature, and propose 10 research opportunities in this area.

show abstract

Section: O1mentioning

confidence: 99%

Section: O1mentioning

confidence: 99%

Maintaining behaviour driven development specifications: Challenges and opportunities

Binamungu

Embury

Κωνσταντίνου

2018

2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

show abstract

“…There are different views associated with a set of attributes proposed to measure the quality of unit test cases in the literature [9]. Kaczanowski [25] discusses three characteristics to measure test quality: Test smells, code coverage and mutation score indicator.…”

Section: Unit Testingmentioning

confidence: 99%

On the effectiveness of unit tests in test-driven development

Tosun

Ahmed

Turhan

et al. 2018

Proceedings of the 2018 International Conference on Software and System Process

Self Cite

View full text Add to dashboard Cite

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice.Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature.Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.

show abstract

“…Table 3 shows the metrics which our tool recorded. We adapted these metrics from the testing principles defined in our previous work [7]. These metrics capture multiple facets of test quality and have been repeatedly used in the literature for the purpose of determining test quality.…”

Section: Data Extraction For Test Effectivenessmentioning

confidence: 99%

“…In our approach we establish whether there are any characteristics that distinguish effective from non-effective JUnit tests by identifying a set of test code metrics based on existing testing principles [7]. We develop a tool to collect this set of metrics from the JUnit tests in seven open source Java systems.…”

Section: Introductionmentioning

confidence: 99%

How Effectively Is Defective Code Actually Tested?

Petrić

Hall

Bowes

2018

Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering

Self Cite

View full text Add to dashboard Cite

Background: Newspaper headlines still regularly report latent software defects. Such defects have often evaded testing for many years. It remains difficult to identify how well a system has been tested. It also remains difficult to assess how successful at finding defects particular tests are. Coverage and mutation testing are frequently used to asses test effectiveness. We look more deeply at the performance of commonly used JUnit testing by assessing how much JUnit testing was done and how effective that testing was at detecting defects in seven open source systems. Aim: We aim to identify whether defective code has been effectively tested by JUnit tests as non-defective code. We also aim to identify the characteristics of JUnit tests that are related to identifying defects. Methodology: We first extract the defects from seven open source projects using the SZZ algorithm. We match those defects with JUnit tests to identify the proportion of defects that were covered by JUnit tests. We also do the same for non-defective code. We then use Principal Component Analysis and machine learning to investigate the characteristics of JUnit tests that were successful in identifying defects. Results: Our findings suggest that most of the open source systems we investigated are under-tested. On average over 66% of defective methods were not linked to any JUnit tests. We show that the number of methods touched by a JUnit test is strongly related to that test uncovering a defect. Conclusion: More JUnit tests need to be produced for the seven open source systems that we investigate. JUnit tests need to be relatively sophisticated, in particular they should touch more than just one method during the test.

show abstract

How Good Are My Tests?

Cited by 32 publications

References 32 publications

Maintaining behaviour driven development specifications: Challenges and opportunities

Maintaining behaviour driven development specifications: Challenges and opportunities

On the effectiveness of unit tests in test-driven development

How Effectively Is Defective Code Actually Tested?

Contact Info

Product

Resources

About