CONCORD: Clone-Aware Contrastive Learning for Source Code

Abstract

Deep Learning (DL) models to analyze source code have shown immense promise during the past few years. More recently, selfsupervised pre-training has gained traction for learning generic code representations valuable for many downstream SE tasks, such as clone and bug detection.While previous work successfully learned from different code abstractions (e.g., token, AST, graph), we argue that it is also essential to factor in how developers code day-to-day for learning generalpurpose representation. On the one hand, human developers tend to write repetitive programs referencing existing code snippets from the current codebase or online resources (e.g., Stack Overflow website) rather than implementing functions from scratch; such behaviors result in a vast number of code clones. In contrast, a deviant clone by mistake might trigger malicious program behaviors.Thus, as a proxy to incorporate developers' coding behavior into the pre-training scheme, we propose to include code clones and their deviants. In particular, we propose CONCORD, a self-supervised pre-training strategy to place benign clones closer in the representation space while moving deviants further apart. We show that CONCORD's clone-aware pre-training drastically reduces the need for expensive pre-training resources while improving the performance of downstream SE tasks. We also empirically demonstrate that CONCORD can improve existing pre-trained models to learn better representations that consequently become more efficient in both identifying semantically equivalent programs and differentiating buggy from non-buggy code. CCS CONCEPTS• Software and its engineering → Language features; • Computing methodologies → Knowledge representation and reasoning.

Coca: Improving and Explaining Graph Neural Network-Based Vulnerability Detection Systems

“…To train a detection model robust to random perturbations, we borrow the contrastive learning technique to learn better feature representations. Despite the similarity in terms of the high-level design idea [4,23,34,46], i.e., pre-training a self-supervised featureacquisition model over a large unlabeled code database, and performing supervised fine-tuning over labeled dataset to transfer it to a specific downstream SE task, we employ an additional supervised contrastive loss term to effectively leverage label information.…”