Software Test Capability Improvement Method

Test automation maturity improves product quality—Quantitative study of open source projects using continuous integration

“…This reveals that, increased test automation effort caused by improving the level of test automation maturity and product quality is not evidenced. Our finding conflicts with prior studies' viewpoint [33,28,37,30,31], which states that, at higher levels of test automation maturity and product quality, there is increased effort to develop and execute rigorous automated tests. Our finding confirms the other viewpoint, which states the effort savings may offset the invested effort when using standard best practices to perform rigorous test automation [32,34,35,39].…”

“…This team reported that, the higher level of test automation maturity is associated with shorter release cycles and better product quality in the CI context. [33] 2020 As noted in Table 2, in all studies [27,33,34,35,28,37,38,39,30,32] that have observed how test automation maturity affects product quality, the findings verified the positive impact. These studies view product quality as to which extent a software product free from defects during the quality assurance process or in the production environment.…”

“…It is different from scholars who defined product quality as conformance to a standard or to which extent a software product bears on its ability to satisfy given needs [40,41]. Studies [27,28,37,32,35,39] reported that better product quality can be achieved after improving test automation maturity against certain standard best practices.…”

“…One school of viewpoint states the negative relationship of test automation maturity and product quality with test automation effort. Studies [33,28,37,30,31], which hold this view, have observed the increased effort to develop and execute more rigorous automated tests in attaining higher product quality. Experience study [36] reported that the direct effect of test automation maturity on test automation effort is negative.…”

“…Last, our research aims to validate whether standard best practices in the literature should be recommended for CI practitioners. Among all identified studies, only studies [27,37,28] had the similar aim. Their findings were based on authors' experience.…”