What Norwegian Developers Want and Need From Security-Directed Program Analysis Tools

Sørensen, Elias Brattli; Karlsen, Edvard Kristoffer; Li, Jingyue

doi:10.1145/3383219.3383293

Cited by 2 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applying different confidence rankings can lead to different detection results. We believed that many developers would prefer low false positives [13], [17], [18]. Therefore, we modified its confidence rating as follows.…”

Section: Other Factors That Influenced Plugin Evaluation Resultsmentioning

confidence: 99%

“…Study [1] evaluated five open-source IDE plugins, which use static source code analysis approaches to detect vulnerabilities. The results of [1] show that the plugins have much higher false-negative and false-positive rates than the level (as explained in [17] [13] [18]) that developers are willing to accept. Similar empirical studies, e.g., [19], [20], and [2], also observe low recall and precision of IDE plugins in detecting many vulnerabilities.…”

Section: Methodsmentioning

confidence: 99%

“…Developers like to use vulnerability detectors which report high precision and recall [17] [13] [18]. Our results show that precision and recall reported by the tool vendors can be misleading without proper root cause analyses of the evaluation result.…”

Section: B Implication For Industry Practitionersmentioning

confidence: 99%

See 2 more Smart Citations

Critical Understanding of Security Vulnerability Detection Plugin Evaluation Reports

Beba

Karlsen

et al. 2021

2021 28th Asia-Pacific Software Engineering Conference (APSEC)

Self Cite

View full text Add to dashboard Cite

Integrated development environment (IDE) plugins aimed at detecting web application security vulnerabilities can help developers create secure applications in the first place. Most of such IDE plugins use static source code analysis approaches. Although several empirical studies evaluated the plugins and compared their precision and recall of detecting web application security, few follow-up studies tried to understand the evaluation results. We analyzed more than 20,000 vulnerability reports based on 7,215 distinct test cases spanning 11 categories of web application vulnerabilities to understand the evaluation results of three open-source IDE plugins, namely, SpotBugs, FindSecBugs, and Early Security Vulnerability Detector (ESVD), which aimed at detecting security vulnerabilities of Java-based web applications. Our results identify many factors besides the source code analysis approach that can dramatically bias the detection performance. Based on our insights, we improved the studied plugins. In addition, our study raises the alarm that, without solid root cause analyses, the evaluation and comparisons of security vulnerability detection approaches and tools could be misleading. Thus, we proposed a guideline on reporting the evaluation results of the security vulnerability detection approaches.Index Terms-Software security, Source code analysis, Vulnerability detection, Empirical study, Root cause analysis• First, we provide insights on issues and solutions of implementing taint analysis-based security vulnerability detection tools. Some of the insights can also be generalized to other types of static source code analysis-based vulnerability detectors. • A more significant contribution is that our deep under-

show abstract

Section: Other Factors That Influenced Plugin Evaluation Resultsmentioning

confidence: 99%