Do Android taint analysis tools keep their promises?

Abstract

In recent years, researchers have developed a number of tools to conduct taint analysis of Android applications. While all the respective papers aim at providing a thorough empirical evaluation, comparability is hindered by varying or unclear evaluation targets. Sometimes, the apps used for evaluation are not precisely described. In other cases, authors use an established benchmark but cover it only partially. In yet other cases, the evaluations differ in terms of the data leaks searched for, or lack a ground truth to compare against. All those limitations make it impossible to truly compare the tools based on those published evaluations.We thus present ReproDroid, a framework allowing the accurate comparison of Android taint analysis tools. ReproDroid supports researchers in inferring the ground truth for data leaks in apps, in automatically applying tools to benchmarks, and in evaluating the obtained results. We use ReproDroid to comparatively evaluate on equal grounds the six prominent taint analysis tools Amandroid, DIALDroid, DidFail, DroidSafe, FlowDroid and IccTA. The results are largely positive although four tools violate some promises concerning features and accuracy. Finally, we contribute to the area of unbiased benchmarking with a new and improved version of the open test suite DroidBench.

Web APIs in Android through the Lens of Security

“…Two of the observed URLs were related to advertising distribution services, i.e., http://media.admob.com (283, 2.4%) and https: //pagead2.googlesyndication.com (271, 2.3%). 8 We found that the http URL scheme (7 208 occurrences, 61%) is much more prevalent than its secure counterpart https (4 531 occurrences, 38%). Besides findings of the two common schemes we found few appearances of the ws (WebSocket) protocol (4 occurrences, 0.0%), which provides (unprotected) full-duplex communication on top of HTTP TCP connections.…”

“…2) Detection and extraction: In principal, we need to track flows of data in relevant APIs, and several static analysis frameworks exist to track data flows in Android apps. Nevertheless, in our experience as well as according to recent studies, these tools may not perform as described in the relevant papers [7], [8], [9]. We therefore decided to implement our own lightweight analysis tailored to reconstruct web APIs in the code.…”

BenchPress: Analyzing Android App Vulnerability Benchmark Suites

“…They also established the representativeness of Ghera benchmarks (in terms of API usage) [15]. Pauck et al [13] developed ReproDroid, a tool to help verify the authenticity of Android app vulnerability benchmarks. They found that not all claims about the presence/absence of vulnerabilities in benchmarks in DIALDroid, DroidBench, and ICCBench benchmark suites were true.…”

“…DroidBench [13] contains 211 benchmarks. Each benchmark is an Android app that captures zero or more information leak vulnerabilities.…”

“…While there has been considerable interest in developing solutions for detecting vulnerabilities in Android apps, very few efforts are focused on developing and measuring benchmarks used to evaluate the effectiveness of the solutions. Recently, Pauck et al [13] developed a framework for verifying the authenticity of benchmarks in DroidBench and IccBench and refining them. In a similar vein, Qiu et al [14] discovered that a few benchmarks in DroidBench and IccBench captured multiple aspects, which made their evaluation of the effectiveness of static taint analysis tools difficult.…”