A large scale study of programming languages and code quality in github

Abstract

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.

Studying the evolution of PHP web applications

“…As in other studies (e.g. [44]) we identified bug related commits by filtering those that contain error related keywords, such as 'error', 'bug', 'fix' and 'issue' in the corresponding commit message.…”

Assert Use in GitHub Projects

“…We have recently categorized GitHub projects into six general and disjoint domains, including Databases, Libraries, etc. [40] As code in these different categories may be substantially different [40], it is reasonable to expect that the code development process, including debugging, may be different across these domains. While there we did not find a relationship between code quality and application domain in our prior work, [40], assert use might be related to the domain.…”

“…In our recent work on code defects in the GitHub corpus [40], we categorized projects based on their application domain into six general groups: Applications, Code Analyzer, Database, Framework, Library, and Middleware. To investigate whether the use of asserts depended in anyway on the application domain, we used negative binomial regression due to the smaller sample size (see Methodology), with the domain as a factor, while controlling for the total number of lines, developers, and age of the project.…”

Identifying Metrics' Biases When Measuring or Approximating Size in Heterogeneous Languages

“…Third, we aim at covering the range of metrics that are used to measure or approximate size in other research works, e.g. by Malaiya [4], Nugroho et al [7], or Ray et al [10]. Therefore, we consider metrics that base on different concepts, e.g.…”

“…KLOC as by Nugroho et al [7], but also sometimes approximated, e.g. as number of commits by Ray et al [10].…”