A Security Perspective on Code Review: The Case of Chromium

Abstract

Modern Code Review (MCR) is an established software development process that aims to improve software quality. Although evidence showed that higher levels of review coverage relates to less post-release bugs, it remains unknown the effectiveness of MCR at specifically finding security issues.We present a work we conduct aiming to fill that gap by exploring the MCR process in the Chromium open source project. We manually analyzed large sets of registered (114 cases) and missed (71 cases) security issues by backtracking in the project's issue, review, and code histories. This enabled us to qualify MCR in Chromium from the security perspective from several angles: Are security issues being discussed frequently? What categories of security issues are often missed or found? What characteristics of code reviews appear relevant to the discovery rate?Within the cases we analyzed, MCR in Chromium addresses security issues at a rate of 1% of reviewers' comments. Chromium code reviews mostly tend to miss language-specific issues (e.g., C++ issues and buffer overflows) and domain-specific ones (e.g., such as Cross-Site Scripting); when code reviews address issues, mostly they address those that pertain to the latter type. Initial evidence points to reviews conducted by more than 2 reviewers being more successful at finding security issues.

Why Don’t Developers Detect Improper Input Validation? '; DROP TABLE Papers; --

“…The authors' results indicate relatively low effectiveness in vulnerability detection. In line with these results, di Biase et al [30] found that approximately only 1% of Chromium's review comments are about potential security flaws. While these studies [27]- [30] mainly focus on the effectiveness of code review on vulnerability detection, our study investigates whether developers can(not) detect vulnerabilities (specifically IIV) during code review and the underlying causes with a focus on developers' knowledge and mindset.…”

When testing meets code review

“…Thongtanunam et al [43] evaluated the impact that characteristics of MCR practices have on software quality, studying MCR practices in defective and clean source code files. Di Biase et al [22] analyzed the Chromium system to understand the impact of MCR on finding security issues, showing that the majority of missed security flaws relate to language-specific issues. However, these studies, as well as most of the current literature on contemporary code review, either focus on production files only or do not explicitly differentiate production from test code.…”

Why Security Defects Go Unnoticed During Code Reviews? A Case-Control Study of the Chromium OS Project

“…Moreover, historical evidence provides several examples of individual cases that contributed to discovery in physics, economics, and social science (see "Five misunderstandings about case-study research" by Flyvjerg [20]). Even in the SE domain case studies of the Chromium project [14], [17], Apache case study by Mockus et al [42], and Mozilla case study by Khomh et al [26] have provided important insights. To promote building knowledge through families of experiments, as championed by Basili [5], we have made our dataset and scripts publicly available [48].…”