Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing

Abstract

The number of mobile applications has increased geometrically nowadays, but how to ensure their quality and conduct adequate and effective testing is still a challenge for developers. On the one hand, the number of mobile apps is increasing, and the update speed is faster and faster. Many small and mediumsized companies can hardly test the app adequately before each release. On the other hand, mobile apps play more and more important roles in people's life, such as financial payment. For the sake of company security and user privacy, most companies will encrypt key codes in their APP. Even third-party testers cannot get source code, which also leads to many researchers cannot carry out further research and effective testing for these widely used mainstream APPs. Code coverage is an important indicator to guide software testing, which plays a crucial role in ensuring the quality of testing. However, it is an urgent problem to find accurate coverage indictors to evaluate these tests. And when testing those existing widely used mainstream closed-source apps, we find that the existing coarse-grained coverage metrics like method coverage is bad coverage indictors for app testing that can exaggerate or minimize the actual coverage rate, which cannot obtain satisfactory results for the evaluation of test effects. To find a more reliable coverage indictor, this paper demonstrates the correctness of instruction coverage indictor and the inaccurate of method coverage in evaluating the test of closed-source APP from the perspective of probability and statistics. Then we shows how inaccurate the method coverage can be through an empirical evaluation on datasets of closed source APPs and open source APPs respectively. It is further verified that instruction coverage is a more effective evaluation indictor than methods coverage or activity coverage in the test of closed source APP for the first time. INDEX TERMS Mobile testing, automated testing, coverage criteria, instruction coverage.

MT-Forward Algorithm for Prediction of Project Categories Based on Selection of Mutants

“…Sometimes not all methods are used in application sources. It requires a test on the coverage method [37]. Branch coverage (CB) covers the possible steps in the branching flow control structure followed.…”

DroidbotX: Test Case Generation Tool for Android Applications Using Q-Learning

“…Huang et al [68] first proposed the concept of instruction coverage, which is used in many studies as an indicator to evaluate test efficiency [24,44,54,64]. It is a more accurate and valid test coverage criterion that reflects the adequacy of testing results for closed-source apps [25]. (2) Method coverage is the ratio of the number of methods called during execution of the AUT to the total number of methods in the source code of the app.…”

“…Instruction coverage provides information about the amount of code that has been executed. Hence, improving instruction and method coverage ensures that more of the app's functionalities associated with each activity are explored and tested [23][24][25]. Similarly, activity coverage is a necessary condition to detect crashes that can occur when interacting with the app's UI.…”