CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses

Afrose, Sharmin; Rahaman, Sazzadur; Yao, Danfeng

doi:10.1109/secdev.2019.00017

Cited by 21 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For this dataset, we evaluated the execution times and the number of crypto misuses found by the two tools. The second dataset is the CryptoAPI-Bench [26], a set of Java benchmarks that include crypto misuses. For this dataset, we determined the false positive and the false negative rates of the two tools.…”

Section: Results: Comparison With Cryptoguardmentioning

confidence: 99%

“…one of the most effective static tools to detect misuses: we use 150 popular Android apps of the Google Play Store for the comparison; we show that CRYLOGGER reports misuses that CryptoGuard misses, but we show that the opposite is also possible, thus making the case for combining static and dynamic approaches; 4. we reverse engineer 150 Android apps to evaluate the false positives of CryptoGuard; we show that for some rules many false positives are reported due to insecure, but untriggerable, code included in the apps; 5. we compare CRYLOGGER against CryptoGuard by using the CryptoAPI-Bench [26], a set of Java programs that include misuses; we also extend the CryptoAPI-Bench with tests cases suited for dynamic tools; 6. we use CRYLOGGER to analyze 1780 Android apps downloaded from the Google Play Store (the dataset was collected between September and October 2019). These are the most popular apps of 33 different categories.…”

Section: A Contributionsmentioning

confidence: 99%

“…Recent studies [6], [26] showed that CryptoGuard is one of the most effective tools in reducing the false positives, thanks to rulespecific algorithms that refine the results of the static analysis. We show, however, that CryptoGuard still produces many false positives in practice by reporting crypto misuses that can never be triggered at runtime (Section VIII).…”

Section: A Detection Of Crypto Misusesmentioning

confidence: 99%

See 2 more Smart Citations

CRYLOGGER: Detecting Crypto Misuses Dynamically

Piccolboni,

Di Guglielmo,

Carloni

et al. 2020

Preprint

View full text Add to dashboard Cite

Cryptographic (crypto) algorithms are the essential ingredients of all secure systems: crypto hash functions and encryption algorithms, for example, can guarantee properties such as integrity and confidentiality. Developers, however, can misuse the application programming interfaces (API) of such algorithms by using constant keys and weak passwords. This paper presents CRYLOGGER, the first open-source tool to detect crypto misuses dynamically. CRYLOGGER logs the parameters that are passed to the crypto APIs during the execution and checks their legitimacy offline by using a list of crypto rules. We compare CRYLOGGER with CryptoGuard, one of the most effective static tools to detect crypto misuses. We show that our tool complements the results of CryptoGuard, making the case for combining static and dynamic approaches. We analyze 1780 popular Android apps downloaded from the Google Play Store to show that CRYLOGGER can detect crypto misuses on thousands of apps dynamically and automatically. We reverse-engineer 28 Android apps and confirm the issues flagged by CRYLOGGER. We also disclose the most critical vulnerabilities to app developers and collect their feedback.

show abstract

Section: Results: Comparison With Cryptoguardmentioning

confidence: 99%

Section: A Contributionsmentioning

confidence: 99%

Section: A Detection Of Crypto Misusesmentioning

confidence: 99%

See 1 more Smart Citation

CRYLOGGER: Detecting Crypto Misuses Dynamically

Piccolboni,

Di Guglielmo,

Carloni

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The picked app will be added to P 0 until the results of P are completely covered. Specifically, we developed our mapping with only 198 apps in P 0 (175 from CryptoAPI-Bench [25] and 23 from AndroZoo [26]), which is less than 0.5% of all the app samples.…”

Section: Implementation Settingsmentioning

confidence: 99%

CryptoEval: Evaluating the Risk of Cryptographic Misuses in Android Apps with Data-Flow Analysis

Xu¹,

Wu²,

Zeng³

et al. 2021

Preprint

View full text Add to dashboard Cite

The misunderstanding and incorrect configurations of cryptographic primitives have exposed severe security vulnerabilities to attackers. Due to the pervasiveness and diversity of cryptographic misuses, a comprehensive and accurate understanding of how cryptographic misuses can undermine the security of an Android app is critical to the subsequent mitigation strategies but also challenging. Although various approaches have been proposed to detect cryptographic misuses in Android apps, seldom studies have focused on estimating the security risks introduced by cryptographic misuses. To address this problem, we present an extensible framework for deciding the threat level of cryptographic misuses in Android apps. Firstly, we propose a unified specification for representing cryptographic misuses to make our framework extensible and develop adapters to unify the detection results of the state-of-the-art cryptographic misuse detectors, resulting in an adapter-based detection toolchain for a more comprehensive list of cryptographic misuses. Secondly, we employ a misuse-originating data-flow analysis to connect each cryptographic misuse to a set of data-flow sinks in an app, based on which we propose a quantitative data-flow-driven metric for assessing the overall risk of the app introduced by cryptographic misuses. To make the per-app assessment more useful in the app vetting at the app-store level, we apply unsupervised learning to predict and classify the top risky threats, to guide more efficient subsequent mitigations. In the experiments on an instantiated implementation of the framework, we evaluate the accuracy of our detection and the effect of data-flow-driven risk assessment of our framework. Our empirical study on over 40,000 apps as well as the analysis of popular apps reveals important security observations on the real threats of cryptographic misuses in Android apps.

show abstract

“…A preliminary version of the work appeared in the Proceedings of the 2019 ACM Conference on Computer and Communications Security (CCS) [5] and 2019 IEEE Secure Development Conference (SecDev) [28]. We expanded the conference version by adding a new benchmark ApacheCryptoAPI-Bench (Section 4, Table 2) that contains complex real-world Java programs and we test four static tools' performance in real-world code (Section 6.4, Table 7, Table 8).…”

mentioning

confidence: 99%

Evaluation of Static Vulnerability Detection Tools with Java Cryptographic API Benchmarks

Afrose¹,

Ye²,

Rahaman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. To compare their accuracy and security guarantees, we develop two comprehensive benchmarks named CryptoAPI-Bench and ApacheCryptoAPI-Bench. CryptoAPI-Bench consists of 181 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false-positive rates. The ApacheCryptoAPI-Bench consists of 121 cryptographic cases from 10 Apache projects. We evaluate four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity using both benchmarks. We present their performance and comparative analysis. The ApacheCryptoAPI-Bench also examines the scalability of the tools. Our benchmarks are useful for advancing state-of-the-art solutions in the space of misuse detection.

show abstract

CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses

Cited by 21 publications

References 24 publications

CRYLOGGER: Detecting Crypto Misuses Dynamically

CRYLOGGER: Detecting Crypto Misuses Dynamically

CryptoEval: Evaluating the Risk of Cryptographic Misuses in Android Apps with Data-Flow Analysis

Evaluation of Static Vulnerability Detection Tools with Java Cryptographic API Benchmarks

Contact Info

Product

Resources

About