Continuous Test-Driven Development: A Preliminary Empirical Evaluation Using Agile Experimentation in Industrial Settings

Madeyski, Lech; Kawalerowicz, Marcin

doi:10.1007/978-3-319-65208-5_8

Cited by 9 publications

(12 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both experiments were conduced on two different industry grade software projects. As mentioned before, the results of the first experiment were reported by the authors in [17].…”

Section: Experiments Executionmentioning

confidence: 67%

“…If there is a performance gain due to CTDD then (taking into account the mentioned popularity of TDD among software developers) an organisation using TDD or the entire software industry could observe benefits from switching to CTDD. This paper extends our earlier conference paper [17], including preliminary CTDD evaluation (using single case experimental design) with only one developer, by performing a second CTDD experiment (using small-n experimental design) in a different project involving two additional developers. We also present much more in depth analysis of the results of both experiments using effect size measures (discussed in Section 6.3) to provide more reliable conclusions wrt.…”

Section: Introductionmentioning

confidence: 81%

“…Plan, execution (including the tool set we built to perform the experiments), results, as well as analysis and discussion of the results of the conducted experiments are presented in Section 3, 4, 5, 6, respectively. The results of the first experiment were discussed earlier [17]. The results of the second experiment and detailed analysis of the results of both experiments (including four effect size measures) are presented in this paper for the first time.…”

Section: Introductionmentioning

confidence: 89%

“…Using single-case/small-n experimental design combined with randomization tests [7] (that are the core elements behind our idea of Agile Experimentation in software engineering [16]) were until now mainly used in the social psychology, medicine, education, rehabilitation, and social work [10], although there were first inroads into using those techniques in software engineering as well [8,16,17,25].…”

Section: Experimental Designmentioning

confidence: 99%

See 3 more Smart Citations

Empirical evaluation of continuous test-driven development in industrial settings

Madeyski

Kawalerowicz

2019

IFS

Self Cite

View full text Add to dashboard Cite

BACKGROUND: Continuous Test-Driven Development (CTDD) is, proposed by the authors, enhancement of the wellestablished Test-Driven Development (TDD) agile software development and design practice. CTDD combines TDD with continuous testing (CT) that essentially perform background testing. The idea is to eliminate the need to execute tests manually by a TDD-inspired developer. OBJECTIVE: The objective is to compare the efficiency of CTDD vs TDD measured by the red-to-green time (RTG time), i.e., time from the moment when the project is rendered not compiling or any of the tests is failing, up until the moment when the project compiles and all the tests are passing. We consider the RTG time to be a possible measurement of efficiency because the shorter the RTG time, the quicker the developer is advancing to the next phase of the TDD cycle. METHOD: We perform single case and small-n experiments in industrial settings presenting how our idea of Agile Experimentation materialise in practice. We analyse professional developers in a real-world software development project employing Microsoft .NET. We extend the contribution presented in our earlier paper by: 1) performing additional experimental evaluation of CTDD and thus collecting additional empirical evidence, 2) giving an extended, detailed example how to use and analyse both a single case and small-n experimental designs to evaluate a new practice (CTDD) in industrial settings taking into account natural constraints one may observe (e.g., a limited number of developers available for research purposes) and presenting how to reach more reliable conclusions using effect size measures, especially PEM and PAND which are more appropriate when data are not normally distributed or there is a large variation between or within phases. RESULTS: We observed reduced variance and trimmed means of the RTG time in CTDD in comparison to TDD. Various effect size measures (including ES, d-index, PEM, and PAND) indicate small, albeit non-zero, effect size due to CTDD. CONCLUSIONS: Eliminating the reoccurring manual task of selecting and executing tests and waiting for the results (by embracing CTDD) may slightly improve the development speed, but this small change on a level of a single developer, multiplied by a number of developers, can potentially lead to savings on the company or industry level.

show abstract

“…Both experiments were conduced on two different industry grade software projects. As mentioned before, the results of the first experiment were reported by the authors in [17].…”

Section: Experiments Executionmentioning

confidence: 67%

Section: Introductionmentioning

confidence: 81%

Section: Introductionmentioning

confidence: 89%

Section: Experimental Designmentioning

confidence: 99%

See 2 more Smart Citations

Empirical evaluation of continuous test-driven development in industrial settings

Madeyski

Kawalerowicz

2019

IFS

Self Cite

View full text Add to dashboard Cite

show abstract

“…The coding philosophy of Test-Driven Development (TDD) translates a modeling component into a coding requirement and from there into specific test cases that must be passed by a code module (Beck, 2003; Madeyski, 2010; Mäkinen and Munch, 2014). Code is written to pass the tests that model the requirements.…”

Section: Moving Forwardmentioning

confidence: 99%

Issues in Reproducible Simulation Research

2018

Bull Math Biol

View full text Add to dashboard Cite

In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86-89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142-151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation-DEVS integrative M&S symposium, pp 6-11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.

show abstract

A resilience‐based framework for assessing the evolution of open source software projects

Kritikos

Stamelos

2023

J Software Evolu Process

View full text Add to dashboard Cite

Open source software (OSS) has been developing for more than two decades. It originated as a movement with the introduction of the first free/libre OSS operating system, became a popular trend among the developer community, led to enterprise solutions widely embraced by the global market, and began garnering attention from significant players in the software industry (such as IBM's acquisition of RedHat). Throughout the years, numerous software assessment models have been suggested, some of which were created specifically for OSS projects. Most of these assessment models focus on software quality and maintainability. Some models are taking under consideration health aspects of OSS projects. Despite the multitude of these models, there is yet to be a universally accepted model for assessing OSS projects. In this work, we aim to adapt the City Resilience Framework (CRF) for use in OSS projects to establish a strong theoretical foundation for OSS evaluation focusing on the project's resilience as it evolves over time. We would like to highlight that our goal with the proposed assessment model is not to compare two OSS solutions with each other, in terms of resilience, or even do a resilience ranking between the available OSS tools. We are aiming to investigate resilience of an OSS project as it evolves and identify possible opportunities of improvements in the four dimensions we are defining. These dimensions are as follows: source code, business and legal, integration and reuse, and social (community). The CRF is a framework that was introduced to measure urban resilience and most specifically how cities' resilience is changing as they evolve. We believe that a software evaluation model that focuses on resilience can complement the pre‐existing models based on software quality and software health. Although concepts that are related to resilience, like sustainability or viability, already appear in literature, to our best knowledge, there is no OSS assessment model that evaluates the resilience of an OSS project. We argue that cities and OSS projects are both dynamically evolving systems with similar characteristics. The proposed framework utilizes both quantitative and qualitative indicators, which is viewed as an advantage. Lastly, we would like to emphasize that the framework has been tested on the enterprise software domain as part of this study, evaluating five major versions of six OSS projects, Laravel, Composer, PHPMyAdmin, OKApi, PatternalPHP, and PHPExcel, the first three of which are intuitively considered resilient and the three latter nonresilient, to provide a preliminary validation of the models' ability to distinguish between resilient and not resilient projects.

show abstract

Continuous Test-Driven Development: A Preliminary Empirical Evaluation Using Agile Experimentation in Industrial Settings

Cited by 9 publications

References 12 publications

Empirical evaluation of continuous test-driven development in industrial settings

Empirical evaluation of continuous test-driven development in industrial settings

Issues in Reproducible Simulation Research

A resilience‐based framework for assessing the evolution of open source software projects

Contact Info

Product

Resources

About