An Empirical Study Investigating the Predictors of Software Metric Correlation in Application Code and Test Code.

Abstract

On non-trivial software, the large test code base needs adequate maintenance similarly to the application code. It has often been argued that test code should be simple, some authors even arguing that a test script should be reduced to a single control flow. If this is indeed what happens in practice, then we believe test code maintenance will be very different from application code maintenance, despite the fact that both are source code. Using a number of large open source software, we compare application code and test code by using a series of well-known source code complexity metrics. The results reveal that application code is, as expected, more complex than test code but not necessarily so much more complex. The study also confirms the assertion that test code is not as simple as it should be, at least as advocated in textbooks, and may therefore be very complex to maintain. Using complexity metrics to determine the difference, our findings also reveal that, the kind of code determines the extent of monotonicity between the number of lines of code (LOC) and cyclomatic complexity (CC). While a number of authors hypothesize and experimentally confirm that CC has a very strong correlation with LOC, justifying the use of LOC in place of CC (and Halstead Effort), we believe that this strong correlation is prevalent only in production code as results from test code prove otherwise. In test code, there exist a very weak (or almost no) correlation between (a) LOC and CC, (b) Halstead Effort and CC, and (c) LOC and Halstead Effort. We therefore argue that the level of correlation depends on at least three factors namely: the kind of code, the kind of software and the kind of metric. Therefore, we believe it would be inappropriate, without considering these factors to substitute one metric with another using some generalized correlation hypothesis. The results gained thus far contradict the notion that some metrics are correlated (that is if the right factors are not considered). Given the weak monotonicity between CC and LOC, CC and Halstead Effort for test code, we disagree that CC and Halstead metrics are redundant as some studies suggest. Rather, we advocate for the use of CC over LOC (or both or cylomatic density) due to numerous advantages CC has over LOC such as the fact that CC is perceived to better reflect cognitive complexity, numerical complexity, adequacy of testing, interdependency and code refactoring that cannot be accounted for simply by LOC.