An Empirical Evaluation of Search Algorithms for App Testing

Sell, Leon; Auer, Michael; Frädrich, Christoph; Gruber, Michael; Werli, Philemon

doi:10.1007/978-3-030-31280-0_8

Cited by 14 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In summary, this means that EAs are not contributing substantially to gain better statement coverage in Android test generation. This result is also consistent with the study presented by Sell et al 10 for Android test generation in which single-objective and multiobjective algorithms do not perform better than random algorithms, and sometimes they even perform slightly worse.…”

Section: Rq2: What Is the Contribution Of The Nsga-ii Ea In Sapienz C...supporting

confidence: 92%

“…In particular, for Android test generation, in order to obtain statement coverage for a given individual, evolutionary approaches need to push the test case to a device/emulator, start the application, run test case, gather fitness information, and pull it from device/emulator. In their work, 10 Sell et al suggested that high execution costs hamper any meaningful evolution for search algorithms. In our study, we observed that the fitness evaluation might take up to 60 s for a test case, depending on its length.…”

Section: Empirical Studymentioning

confidence: 99%

“…In particular, we are interested in comparing different choices of Evolutionary Algorithms (EAs) for Android test generation. Sell et al 10 presented a study comparing different algorithms for Android test generation, but these algorithms are evaluated on the testing tool MATE 11 . Among other differences, MATE differs from Sapienz in that it uses a widget‐based representation of individuals.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Empirical Study on How Sapienz Achieves Coverage and Crash Detection

Arcuschin

Galeotti

Garbervetsky

2021

J Software Evolu Process

View full text Add to dashboard Cite

Several tools for automatically testing Android applications have been proposed. In particular, Sapienz is a search‐based tool that has been recently deployed in an industrial setting. Although it has been shown that Sapienz outperforms several state‐of‐the‐art tools, it is still to be seen what features of SAPIENZ impact the most on its effectiveness. We conducted an extensive empirical study where we compare the impact of the search algorithm and the usage of motif genes, a more compact representation of individuals. Our empirical study shows that the usage of motif genes improves coverage both for Evolutionary Algorithms and random approaches. In particular, it also shows that NSGA‐II, the multi‐objective evolutionary algorithm used by Sapienz, does not have a clear improvement over other algorithms. In terms of number of crashes detected, our study shows that both NSGA‐II and Random Search perform similarly. While the usage of motif genes improves the crash detection of algorithms, it is not enough to make it statistically significant. These facts cast doubts about the use of Evolutionary Algorithms in the context of Android test generation and suggest that motif genes can have a great impact on the overall effectiveness.

show abstract

Section: Rq2: What Is the Contribution Of The Nsga-ii Ea In Sapienz C...supporting

confidence: 92%

Section: Empirical Studymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Study on How Sapienz Achieves Coverage and Crash Detection

Arcuschin

Galeotti

Garbervetsky

2021

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…Dynodroid [31] extends the random selection using weights and frequencies of events. Modeland search-based strategies such as COBWEB [39], SwiftHand [40], EHB-Droid [41], PUMA [42], EvoDroid [43], DroidBot [44], MobiGUITAR [45], juGULAR [46], ABE [47], Humanoid [48], Sapienz [1], Stoat [32], and MATE [9] apply model-based testing with dedicated search strategies to mobile applications. The type of model and search strategy is different in each approach and can range from simple to sophisticated solutions.…”

Section: Related Workmentioning

confidence: 99%

“…In this context, trial-and-error experiments to empirically determine a suitable configuration of NSGA-II was not a viable option due to the high costs of executing search-based test generation approaches for apps (cf. [7,9]). In contrast, a fitness landscape analysis generally aims for an analytical understanding of the search problem that should help to improve the heuristic.…”

Section: Introductionmentioning

confidence: 99%

A comprehensive empirical evaluation of generating test suites for mobile applications with diversity

Vogel¹,

Tran²,

Grunske³

2021

Information and Software Technology

View full text Add to dashboard Cite

Context: In search-based software engineering we often use popular heuristics with default configurations, which typically lead to suboptimal results, or we perform experiments to identify configurations on a trial-and-error basis, which may lead to better results for a specific problem. We consider the problem of generating test suites for mobile applications (apps) and rely on Sapienz, a state-of-the-art approach to this problem that uses a popular heuristic (NSGA-II) with a default configuration. Objective: We want to achieve better results in generating test suites with Sapienz while avoiding trial-and-error experiments to identify a more suitable configuration of Sapienz. Method: We conducted a fitness landscape analysis of Sapienz to analytically understand the search problem, which allowed us to make informed decisions about the heuristic and configuration of Sapienz when developing Sapienz div. We comprehensively evaluated Sapienz div in a headto-head comparison with Sapienz on 34 apps. Results: Analyzing the fitness landscape of Sapienz, we observed a lack of diversity of the evolved test suites and a stagnation of the search after 25 generations. Sapienz div realizes mechanisms that preserve the diversity of the test suites being evolved. The evaluation showed that Sapienz div achieves better or at least similar test results than Sapienz concerning coverage and the number of revealed faults. However, Sapienz div typically produces longer test sequences and requires more execution time than Sapienz. Conclusions: The understanding of the search problem obtained by the fitness landscape analysis helped us to find a more suitable configuration of Sapienz without trial-and-error experiments. By promoting diversity of test suites during the search, improved or at least similar test results in terms of faults and coverage can be achieved.

show abstract

Improving Search-Based Android Test Generation Using Surrogate Models

Auer

Adler²

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Empirical Evaluation of Search Algorithms for App Testing

Cited by 14 publications

References 18 publications

An Empirical Study on How Sapienz Achieves Coverage and Crash Detection

An Empirical Study on How Sapienz Achieves Coverage and Crash Detection

A comprehensive empirical evaluation of generating test suites for mobile applications with diversity

Improving Search-Based Android Test Generation Using Surrogate Models

Contact Info

Product

Resources

About