Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites

Abstract

Abstract. Pair programming is regarded as one of the practices that can make testing more rigorous, thorough and effective. Therefore, we examined pair programming vs. solo programming with respect to both, thoroughness and fault detection effectiveness of test suites. Branch coverage and mutation score indicator were used as measures of how thoroughly tests exercise programs, and how effective they are, respectively. It turned out that the pair programming practice did not significantly affect branch coverage (U = 471.5, non-significant, r = −.03), and mutation score indicator (U = 422.0, non-significant, r = −.12). These results are consistent with the results of selective analysis in which projects with a limited number of assertions are excluded. Analysis of covariance was performed to get a more sensitive measure of our experiment effect as well as to reduce pre-existing differences among subjects. The obtained results do not support anecdotal opinion regarding the positive impact of pair programming on thoroughness or fault detection effectiveness of unit tests. The validity of the results must be considered within the context of the limitations of the study e.g. it is possible that the benefits of pair programming will appear in longer, more complex projects.

The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment

“…According to Maxwell and Delaney, pre-existing differences among subjects are at least as important a predictor of their scores on the dependent variables as any independent variable [50]. Also, the existing research concerning the impact of pair programming (another software development practice of eXtreme Programming methodology) on the thoroughness and the fault detection effectiveness of unit tests shows that the development method accounts for a relatively small proportion of the total variance in the dependent variables [51,52]. Therefore, it seems justified to take into account also the other variables (shown in Section 4.2) that are not part of the main experimental manipulation (described in Section 3.1) but have an influence on the dependent variables (presented in Sections 3.2 and 3.3).…”

“…Determining which mutants are equivalent to the original program is such a tedious and error-prone activity, that even ignoring equivalent mutants is sometimes considered [63]. That means that we are ready to accept the lower bound on mutation score, called the mutation score indicator [51,52]. Accepting it results in the cost-effective application of a mutation analysis, while it still provides meaningful information about the fault detection effectiveness of test suites.…”

“…Therefore, a new helpful instrument [51,52,64] (i.e. Judy mutation testing tool) was used, which supports generation, compilation and execution (i.e.…”

“…EAM / EMM (accessor / modifier method change), and JTD / JTI mutation operators (this keyword deletion / insertion) [68] were added, because there was no reason to exclude these operators, as there was still no determined set of selective mutation operators for class mutation operators [69]. As a result, the mutation operators set, used in our analysis, consists of the 16 aforementioned mutation operators supported by Judy [52,64]. Judy makes use of an aspect-oriented approach to speed up mutation testing.…”

“…They were available to the subjects via a dedicated web site and assigned SVN accounts. BC and M SI were collected by means of Clover [59] and Judy [51,52,64] tools, respectively. In order to enable better measurement of the development process, especially how development techniques were carried out (e.g.…”