Is Assertion Roulette still a test smell? An experiment from the perspective of testing education

Abstract

Test smells are commonly perceived as having a negative impact on software maintainability and correctness. Research has shown that Assertion Roulette is the most pervasive smell in industrial and open-source systems. However, some recent studies argue that the impact of Assertion Roulette is not as severe as previously believed, and developers usually consider it acceptable.The controversy over the impact of Assertion Roulette also exists in the area of testing education. To assess the impact of Assertion Roulette, we conducted a controlled empirical study with 42 CS students. We recruited participants from two populations, CS1 and a graduate testing course, to see what role experience may have in terms of this test smell's impact. Participants were tasked with implementing a project in Java that passes provided JUnit tests. Through analysis of student-authored source code, we measured the impact of Assertion Roulette using code quality measures and testing behavior measures. Our findings show that the impact of Assertion Roulette on students in this study was minimal. Though students with exposure to the test smell began testing significantly later, they performed similarly in terms of programming quality measures. Thus, it would seem the Assertion Roulette smell is no longer a smell at all, even for less experienced populations like students.

Test Smell Detection Tools: A Systematic Mapping Study

“…Therefore, the investigation of the multiple asserts seems to increase the debugging time, and therefore, the Assertion Roulette is negatively impacting the students' debugging process. Our observation brings an alternate perspective with respect to the recent findings of Bai et al [7], who conjectures that Assertion Roulette may no longer be considered a code smell. In fact, Bai et al [7] demonstrated, through a controlled classroom experiment, that Assertion Roulette does impact neither the frequency of testing nor the accuracy of the test cases.…”

“…Our observation brings an alternate perspective with respect to the recent findings of Bai et al [7], who conjectures that Assertion Roulette may no longer be considered a code smell. In fact, Bai et al [7] demonstrated, through a controlled classroom experiment, that Assertion Roulette does impact neither the frequency of testing nor the accuracy of the test cases. We argue that Assertion Roulette is a program comprehension problem.…”

“…Therefore, the findings from our study are more closely aligned with the actual impact of test smells on code comprehension. Furthermore, this study also allows us to compare our findings against the work by Bai et al [7], which states that Assertion Roulette should not be considered a bad smell for students.…”

“…Unfortunately, there has been a lack of focus on the importance of test smells in the classroom. In a study similar to ours, Bai et al [7] examined how the Assertion Roulette smell affects students' productivity and conduct while writing code. The authors employed the Bowling Score Keeper project, and the student was tasked with writing a Java app to calculate the score of a single bowling game according to a set of specifications and JUnit tests.…”

“…Students were asked to complete the two sessions to obtain extra credit, but they were given the option to choose to have their data examined as part of this study. Based on that consent, out of 56 participants from semester one and 65 from semester two, 7 Web-based learning management system. https://instructure.com/canvas we collected data from 45 participants in the first semester and 51 participants in the second semester.…”