A Code Centric Evaluation of C/C++ Vulnerability Datasets for Deep Learning Based Vulnerability Detection Techniques

Jain, Ridhi; Gervasoni, Nicole; Ndhlovu, Mthandazo; Rawat, Sanjay

doi:10.1145/3578527.3578530

Cited by 13 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, deep learning is also used to generate pseudo-code from program code [86], classify code according to readability [87], or summarise code [88]. In the same context, deep learning is also used to detect code vulnerability [89,90]-Technical depth and code smell detection Machine learning has recently been often used to detect self-admitted technical depth [91][92][93] or technical debt in general [94,95]. Technical debt is closely linked to code smells and anti-patterns, which are spotted [96,97], classified [98,99], or identified [100]with artificial intelligence techniques.…”

Section: Deep Learning In Empirical Software Engineering Focusing On ...mentioning

confidence: 99%

The Use of AI in Software Engineering: Synthetic Knowledge Synthesis of Recent Research Literature

Kokol

2024

Preprint

View full text Add to dashboard Cite

Artificial intelligence (AI) has witnessed an exponential increase in its use in various applications. Recently, the academic community started to research and inject new AI-based approaches to provide solutions to traditional software engineering problems. However, a comprehensive and holistic understanding of the current status is missing. To close the above gap, synthetic knowledge synthesis was used to induce a research landscape of the contemporary research literature on the use of AI in software engineering. The synthesis resulted in 15 research categories and five themes, namely natural language processing in software engineering, use of artificial intelligence in the management of software development life cycle, use of machine learning in fault/defect prediction and effort estimation, employment of deep learning in intelligent software engineering and code management, and mining software repositories to improve software quality. The most productive country was China (n=2042), followed by the United States (n=1193), India (n=934), Germany (n=445), and Canada (n=381). A high percentage (n=47.4%) of papers were funded, showing a strong interest in this research topic. The convergence of AI and software engineering can significantly reduce needed resources, improve quality, increase user experience, and improve the well-being of software developers.

show abstract

Section: Deep Learning In Empirical Software Engineering Focusing On ...mentioning

confidence: 99%

The Use of AI in Software Engineering: Synthetic Knowledge Synthesis of Recent Research Literature

Kokol

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly, deep learning is also used to generate pseudo-code from program code [77], classify code according to readability [78], or summarize code [79]. In the same context, deep learning is also used to detect code vulnerability [80,81].…”

Section: Deep Learning In Empirical Software Engineering Focusing On ...mentioning

confidence: 99%

The Use of AI in Software Engineering: A Synthetic Knowledge Synthesis of the Recent Research Literature

Kokol

2024

Information

View full text Add to dashboard Cite

Artificial intelligence (AI) has witnessed an exponential increase in use in various applications. Recently, the academic community started to research and inject new AI-based approaches to provide solutions to traditional software-engineering problems. However, a comprehensive and holistic understanding of the current status needs to be included. To close the above gap, synthetic knowledge synthesis was used to induce the research landscape of the contemporary research literature on the use of AI in software engineering. The synthesis resulted in 15 research categories and 5 themes—namely, natural language processing in software engineering, use of artificial intelligence in the management of the software development life cycle, use of machine learning in fault/defect prediction and effort estimation, employment of deep learning in intelligent software engineering and code management, and mining software repositories to improve software quality. The most productive country was China (n = 2042), followed by the United States (n = 1193), India (n = 934), Germany (n = 445), and Canada (n = 381). A high percentage (n = 47.4%) of papers were funded, showing the strong interest in this research topic. The convergence of AI and software engineering can significantly reduce the required resources, improve the quality, enhance the user experience, and improve the well-being of software developers.

show abstract

“…In recent years, many research works have attempted to fine-tune Large Language Models (LLMs) on source code for SVD, as will be described in Section 2. However, the results that seem most promising are achieved using synthetic datasets, which do not reflect fully the complex nature of real-world source code [12]. As a result, these models tend to perform poorly on real-world data.…”

Section: Introductionmentioning

confidence: 98%

Large Language Models for Source Code Vulnerability Detection: A DiverseVul Analysis

Dias,

Gonçalves,

Maia

et al. 2024

Preprint

View full text Add to dashboard Cite

Software's pervasive impact and increasing reliance in the era of digital transformation raises concerns about vulnerabilities. The emergence of code assistants for code generation further aggravates this issue, emphasizing the critical need for prioritizing software security. In recent research, DL has become a very promising instrument for vulnerability detection, with most recent approaches utilizing LLMs for Software Vulnerability Detection. The data utilized in this context is curated from real-world projects or synthetically, with the latter achieving great metric scores but poor performance in a real context. Driven by this issue, DiverseVul has been curated to be the largest dataset containing C/C++ vulnerable and non-vulnerable functions extracted from real-world projects. This work intends to explore this dataset by using it to fine-tune three LLMs (i.e.: CodeBERT, CodeGPT, and NatGen), evaluating their performance on vulnerability detection. During data processing, several erroneous data points were found, motivating the creation of a refined version of the dataset. Moreover, to establish a comparable baseline, the same models were fine-tuned on the RCVEFixes dataset, which is a refined version of the CVEFixes dataset containing only C/C++ functions. The results show that the best-performing models were CodeBERT trained on DiverseVul with 69% F1-Score and NatGen trained on RVCEFixes with 53%. It can be concluded that the performance of CodeBERT trained on DiverseVul is generally higher than average literature using similar techniques.

show abstract

A Code Centric Evaluation of C/C++ Vulnerability Datasets for Deep Learning Based Vulnerability Detection Techniques

Cited by 13 publications

References 15 publications

The Use of AI in Software Engineering: Synthetic Knowledge Synthesis of Recent Research Literature

The Use of AI in Software Engineering: Synthetic Knowledge Synthesis of Recent Research Literature

The Use of AI in Software Engineering: A Synthetic Knowledge Synthesis of the Recent Research Literature

Large Language Models for Source Code Vulnerability Detection: A DiverseVul Analysis

Contact Info

Product

Resources

About