An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites

Abstract

Randoop is a well-known tool that proposes a feedbackdirected algorithm for automatic and random generation of unit tests for a given Java class. It automatically generates two test suites for the class under test: 1) an error-revealing test suite, and 2) a regression test suite. Despite successful experiences with applying Randoop on real world projects like Java Development Kit (JDK) which have led to creation of error-revealing tests and identification of real bugs, it has not been investigated in the literature how useful are the regression test suites generated by Randoop. In this paper, we have investigated flakiness and fragility of Randoop's regression tests during evolution of 5 open source Java projects with a total of 78 versions. The results demonstrate that the flakiness of the regression tests is not generally noticeable, since in our dataset, only 5% of the classes have at least one flaky regression tests. In addition, test fragility analysis reveals that in most versions of the projects under study, the regression tests generated by Randoop could be successfully executed on many of later versions. Actually, for 2 out of 5 projects in the experiments, the regression tests generated for each version could be successfully executed on all the later versions of the project.

Peeler: Learning to Effectively Predict Flakiness without Running Tests

“…Microsoft deemed flaky tests as one of the most important reasons that impede software deployment period [4]. Beyond these industrial cases, flaky tests were also found to appear among automatically-generated tests by famous tools such as Randoop [5], Evosuite [6], etc.…”

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

“…In the same year, a study on 211 projects in Apache Software Foundation showed that 21% of false alarms (false fail results) are created by flaky tests 7 . Another research work published in 2019 reports 5% flaky tests in Randoop (Obtained from five open‐source Java projects with a total of 78 versions) 8 . Moreover, Microsoft confirmed flaky tests to be the second most important reason (out of 10 reasons) adversely affecting software deployment time 9 .…”