Automated Test Input Generation for Android: Are We There Yet? (E)

Abstract

Mobile applications, often simply called "apps", are increasingly widespread, and we use them daily to perform a number of activities. Like all software, apps must be adequately tested to gain confidence that they behave correctly. Therefore, in recent years, researchers and practitioners alike have begun to investigate ways to automate apps testing. In particular, because of Android's open source nature and its large share of the market, a great deal of research has been performed on input generation techniques for apps that run on the Android operating systems. At this point in time, there are in fact a number of such techniques in the literature, which differ in the way they generate inputs, the strategy they use to explore the behavior of the app under test, and the specific heuristics they use. To better understand the strengths and weaknesses of these existing approaches, and get general insight on ways they could be made more effective, in this paper we perform a thorough comparison of the main existing test input generation tools for Android. In our comparison, we evaluate the effectiveness of these tools, and their corresponding techniques, according to four metrics: code coverage, ability to detect faults, ability to work on multiple platforms, and ease of use. Our results provide a clear picture of the state of the art in input generation for Android apps and identify future research directions that, if suitably investigated, could lead to more effective and efficient testing tools for Android.

Bug Fixes, Improvements, ... and Privacy Leaks - A Longitudinal Study of PII Leaks Across Android App Versions

“…for an app. While a number of other approaches for automation have been proposed, a recent study [18] showed that Monkey exhibited a better coverage in terms of code coverage and fault detection capabilities than other automated tools. Completely random events would prevent apples-to-apples comparison among versions of the same app, so we specify the same random seed that generates the sequence of events for interaction with all of an app's versions.…”

Widget-Sensitive and Back-Stack-Aware GUI Exploration for Testing Android Apps

“…The code coverage can be calculated by analyzing the basic information of byte-code and collecting the runtime information of executed code. We pick two popular automatic testing tools Monkey [5] and Dynodroid [19] for comparison, since a recent research [15] shows that Monkey and Dynodroid achieve higher coverage than other existing testing tools for Android apps. The number of generated events for Monkey is 10000 and for Dynodroid is 2000 (same with what Machiry et al suggested in their work).…”

“…Choudhary et al [15] conduct an empirical study on existing testing tools for Android apps. An interesting discovery is that Monkey and Dynodroid which are based on the random exploration strategy can reach higher coverage than other tools with more sophisticated strategies.…”

“…At last, we traverse the model to generate feasible test cases to cover the user given target. We implemented the proposed techniques into a model based testing tool called AppTag and compared it with two state-of-art tools Monkey [5] and Dynodroid [19] (these tools outperform other existing test generation tools [15]). …”

Why does the orientation change mess up my Android application? From GUI failures to code faults

“…F-Droid offers direct access to app source code and to their developers through code repository and issue tracker. It has been used in many other studies on Android testing proposed in the literature [21,22,23] and contains a growing number of applications belonging to different categories. ‡ https://f-droid.org/ 10 of 27 AMALFITANO ET AL.…”