Flakiness Goes Live: Insights from Running Flaky Tests into the Field

Barboni, Morena; Bertolino, Antonia; Angelis, Guglielmo De

doi:10.2139/ssrn.4095579

Cited by 1 publication

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also plan to extend Groucho in order to collect in vivo execution states that are deemed interesting for developers (e.g., because they trigger a failure), so as to simplify in‐house reproduction of test executions performed in the field. More generally, and in line with some other recent activities [34], an interesting future research concerns the study of practical implications entailed by in vivo testing on the software development process. Even though some of the potential adoption costs have been mitigated by our work (i.e., see Section 4), further research would be needed on the qualitative and quantitative analysis of the most prevalent practices followed by software projects that implement in vivo testing.…”

Section: Discussionmentioning

confidence: 61%

In vivo test and rollback of Java applications as they are

Bertolino

Angelis

Miranda

et al. 2023

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

SummaryModern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. The approach can be transparently applied to the original application, even if only available as compiled code, and it is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence of faults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.

show abstract

Section: Discussionmentioning

confidence: 61%