A comparative study of vulnerability reporting by software composition analysis tools

Abstract

Background: Modern software uses many third-party libraries and frameworks as dependencies. Known vulnerabilities in these dependencies are a potential security risk. Software composition analysis (SCA) tools, therefore, are being increasingly adopted by practitioners to keep track of vulnerable dependencies. Aim: The goal of this study is to understand the difference in vulnerability reporting by various SCA tools. Understanding if and how existing SCA tools differ in their analysis may help security practitioners to choose the right tooling and identify future research needs. Method: We present an in-depth case study by comparing the analysis reports of 9 industry-leading SCA tools on a large web application, OpenMRS, composed of Maven (Java) and npm (JavaScript) projects. Results: We find that the tools vary in their vulnerability reporting. The count of reported vulnerable dependencies ranges from 17 to 332 for Maven and from 32 to 239 for npm projects across the studied tools. Similarly, the count of unique known vulnerabilities reported by the tools ranges from 36 to 313 for Maven and from 45 to 234 for npm projects. Our manual analysis of the tools' results suggest that accuracy of the vulnerability database is a key differentiator for SCA tools. Conclusion: We recommend that practitioners should not rely on any single tool at the present, as that can result in missing known vulnerabilities. We point out two research directions in the SCA space: i) establishing frameworks and metrics to identify false positives for dependency vulnerabilities; and ii) building automation technologies for continuous monitoring of vulnerability data from open source package ecosystems.

Taxonomy of Attacks on Open-Source Software Supply Chains

“…A comparable feature is not present in most compiled languages, like Java, C/C++ or Ruby. In such cases, Integrity check of dependencies through cryptographic hashes [9], [36], [83], [109], [131], [135], [138] 3.3 3.0 2.5 2.0 1.32 Y N 2.3 2.0 Maintain detailed SBOM [5], [8], [53], [183], [184] and perform SCA [8], [31], [43], [48], [51], [53], [55] [42], [123], [185] Code signing [47], [83], [109], [135], [138], [141] Application Security Testing [34], [39], [41], [46], [55], [56], [58], [66], [80], [122], [134], [187] 4 execution is achieved either at runtime, e.g., by embedding the payload in a specific function or initializer, or by poisoning test routines [19]. Differences also exist in regards to code obfuscation and malware detection.…”

On the Effect of Transitivity and Granularity on Vulnerability Propagation in the Maven Ecosystem

“…A number of free and commercial software composition analysis (SCA) tools exist that analyze the open-source components of a project for security risks and license compliance. Each of them differs widely in terms of accuracy, the quality of the vulnerability database, and the level of granularity [33]. For instance, OWASP DC [34] analyzes dependency files of a project and notifies developers if known vulnerabilities are present in the project's transitive dependencies.…”

Are your dependencies code reviewed?: Measuring code review coverage in dependency updates

“…Supply chain security: Recent works have focused on the secure use of open source dependencies as part of the software supply chain [24,31,40,41]. Duan et al have proposed static and dynamic analysis approaches to detect malicious packages for the interpreted languages [19], while Sejfia et al have proposed machine learning models to detect malicious npm packages [33].…”

“…[31,40] Therefore, practitioners are now recommended to review dependency updates before merging them into the codebase [3,39], as the responsibility of security lies on the consumer when using free open-source code [38]. However, manually reviewing all the code changes in each update may not be a practical solution, as projects may have hundreds of direct and transitive dependencies [24,34]. Further, actively maintained packages get frequent updates, overburdening any project that would employ such strict measures.…”