Commit2Vec: Learning Distributed Representations of Code Changes

Abstract

Deep learning methods have found successful applications in fields like image classification and natural language processing. They have recently been applied to source code analysis too, due to the enormous amount of freely available source code (e.g., from open-source software repositories). In this work, we elaborate upon a state-of-the-art approach for source code representation, which uses information about its syntactic structure, and we extend it to represent source code changes (i.e., commits). We use this representation to tackle an industrial-relevant task: the classification of security-relevant commits. We leverage on transfer learning, a machine learning technique which reuses, or transfers, information learned from previous tasks (commonly called pretext tasks) to tackle a new target task. We assess the impact of using two different pretext tasks, for which abundant labeled data is available, to tackle the classification of security-relevant commits. Our results indicate that representations that exploit the structural information in code syntax outperform token-based representations. Furthermore, we show that pre-training on a small dataset (> 10 4 samples), but for a pretext task that is closely related to the target task, results in better performance metrics than pre-training on a loosely related pretext task with a very large dataset (> 10 6 samples).

AI-Assisted Programming Tasks Using Code Embeddings and Transformers

“…Notably, the authors successfully fixed 66 inconsistent method names in a live study on projects in the wild. Cabrera Lozoya et al [100] extended a state-of-the-art approach for representing source code to also include changes in the source code (commits). Transfer learning was then applied to classify security-relevant commits.…”

Co-training for Commit Classification

“…Keshav Ram ( 2020) used a linear SVM on TF-IDF features of Java tokens extracted from code changes, Code2Vec (Alon et al, 2019), convolutional neural network on code changes with surrounding code as context, and BiLSTM (Bi-directional Long Short-Term Memory) on code changes without context to classify commits, remarking that the Code2Vec approach yielded poor results. (Lozoya et al, 2021) proposed a method inspired by Code2Vec to compute vectors from code changes, trained on the pretext task of classifying commits' Jira Ticket Priorities before classifying security-fix commits. Many aforementioned works were conducted under the fullysupervised setting -a setting that can be a stretch for new categories where labeled data is scarce.…”

Fast Changeset-based Bug Localization with BERT

“…Building a semantically rich representations of changesets is relevant to other software engineering applications beyond bug localization, i.e., just-in-time defect prediction, recommendation of a code reviewer for a patch, tangled change prediction. Approaches that define novel changeset embeddings (vector representations of changeset), including CC2Vec [15] and Commit2Vec [28], leverage the difference between added and removed lines of code, among other changeset characteristics. Corley et al [7] studied how including different types of lines from a changeset affects the performance of Latent Dirichlet Allocation-based feature location, observing that including context, additions, and log messages, but excluding removed lines, achieves the best performance.…”