Pizza versus Pinsa: On the Perception and Measurability of Unit Test Code Quality

Abstract

Test cases are an essential asset to evaluate software quality. The research community has provided various alternatives to help developers assessing the quality of tests, like code or mutation coverage. Despite the effort spent so far, however, little is known on how practitioners perceive unit test code quality and whether the existing metrics reflect their perception. This paper aims at addressing this gap of knowledge. We first conduct semi-structured interviews and surveys with practitioners to establish a taxonomy of relevant factors for unit test quality and collect a dataset of tests rated by developers based on their perceived quality. Then, we devise a statistical model to measure how the metrics available in literature reflect the perceived quality of test cases. The findings of our study show that readability and maintainability are the key aspects for developers to diagnose the outcome of test cases and drive debugging activities. On the contrary, code coverage metrics are necessary but not sufficient to evaluate the capability of tests. Finally, we discover that available metrics are effective in characterizing poor-quality tests, while limited in distinguishing high-quality ones.

Developer-centric test amplification

“…After making edits to a test case, one participant wondered whether the original intent of the generated case was still preserved (O#31, Support: 1). This is in line with Grano et al's results: developers are concerned with determining whether a unit test "actually exercises the corresponding unit" and how many relevant scenarios are covered (Grano et al 2020). Prado and Vincenzi showed that the code of a test case is one of the main sources of information about a test case for the developer (Prado and Vincenzi 2018), an observation corroborated by Aniche et al (Aniche et al 2021).…”

“…0-dev-frien dly 3 https:// github. com/ STAMP-proje ct/ dspot/ pull/ 993 to understand for a developer (Grano et al 2020). To make the generated test case easier to understand for the developer, we focus on one input modification at a time (C#3) and add an explanatory comment to every mutation (C#4).…”

Developer-Centric Test Amplification The Interplay Between Automatic Generation and Human Exploration

“…They wished for a separate report of the original instruction coverage and the improvement of instruction coverage after including the amplified test cases. We also observed two times that the developers used the coverage of a test case to infer its intent (O 37, Support: 2), an observation also made by Grano et al (Grano et al, 2020). Sometimes it was unclear to our participants why the new code covers these additional instructions (O 38,Support: 3).…”

“…Our developer-centric amplification leverages the powerful input mutations of DSpot. However, from Grano et al we know that the more complex a test case is, the harder it is to understand for a developer (Grano et al, 2020). To make the generated test case easier to understand for the developer, we focus on one input modification at a time (C 3) and add an explanatory comment to every mutation (C 4).…”

Small-Amp: Test Amplification in a Dynamically Typed Language

“…Involving readability metrics. Based on a previous study (Grano et al 2020), the most important aspects for developers in assessing the quality of a test suite are readability and maintainability. Although the code coverage metrics are necessary for identifying the poor test suites, they are limited in distinguishing high-quality tests based on how practitioners perceive the test quality.…”