Benchmarking Large Language Models for Automated Verilog RTL Code Generation

Thakur, Shailja; Ahmad, Baleegh; Fan, Zhenxing; Pearce, Hammond; Tan, Benjamin; Karri, Ramesh; Dolan-Gavitt, Brendan; Garg, Siddharth

doi:10.23919/date56975.2023.10137086

Cited by 39 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since then, significant progress has been made with the development of larger language models. [9], [10], [14], [15] have been actively investigating the generation of Verilog code at various levels. A milestone was achieved by Blocklove et al [8], who achieved the entire Verilog code generation process using LLMs, resulting in the successful tape out of a microprocessor using a Skywater 130nm shuttle.…”

Section: A Llms On Hardware Description Language (Hdl)mentioning

confidence: 99%

“…BlockLove et al [8] chatted with an LLM and let it design an 8-bit accumulator-based microprocessor. Shailja et al [9] developed a training corpus specifically for Verilog code, which was subsequently utilized to train five distinct fine-tuning LLMs. He et al [10] developed ChatEDA, an interfacing framework and methodology for electronic design automation (EDA).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LADAC: Large Language Model-driven Auto-Designer for Analog Circuits

Liu,

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

Section: A Llms On Hardware Description Language (Hdl)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

LADAC: Large Language Model-driven Auto-Designer for Analog Circuits

Liu,

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…We remark that, this section is focused on verifying LLMs. For the other direction of utilising LLMs to support the verification, there are works related to e.g., specification autoformalisation (Wu et al 2022a), code generation (Thakur et al 2023), assertion generation (Kande et al 2023), zero-shot vulnerability repair (Pearce et al 2023).…”

Section: Verificationmentioning

confidence: 99%

A survey of safety and trustworthiness of large language models through the lens of verification and validation

Huang,

Ruan,

Huang

et al. 2024

Artif Intell Rev

View full text Add to dashboard Cite

Large language models (LLMs) have exploded a new heatwave of AI for their ability to engage end-users in human-level conversations with detailed and articulate answers across many knowledge domains. In response to their fast adoption in many industrial applications, this survey concerns their safety and trustworthiness. First, we review known vulnerabilities and limitations of the LLMs, categorising them into inherent issues, attacks, and unintended bugs. Then, we consider if and how the Verification and Validation (V&V) techniques, which have been widely developed for traditional software and deep learning models such as convolutional neural networks as independent processes to check the alignment of their implementations against the specifications, can be integrated and further extended throughout the lifecycle of the LLMs to provide rigorous analysis to the safety and trustworthiness of LLMs and their applications. Specifically, we consider four complementary techniques: falsification and evaluation, verification, runtime monitoring, and regulations and ethical use. In total, 370+ references are considered to support the quick understanding of the safety and trustworthiness issues from the perspective of V&V. While intensive research has been conducted to identify the safety and trustworthiness issues, rigorous yet practical methods are called for to ensure the alignment of LLMs with safety and trustworthiness requirements.

show abstract

“…This research contributes to the field by providing a systematic approach to iteratively refine code generation tasks, leveraging the capabilities of LLMs to address complex programming challenges effectively. The study conducted by Thakur (2023) [4]. yields significant insights into the capabilities of Large Language Models (LLMs) in generating Verilog code, a fundamental aspect of automating hardware design.…”

Section: Literature Surveymentioning

confidence: 99%

Benchmarking Large Language Models for Code Generation

et al. 2024

IJFMR

View full text Add to dashboard Cite

As the landscape of software development continues to evolve, the need for efficient and innovative coding practices becomes increasingly apparent. This research endeavors to explore the effectiveness of Large Language Models (LLMs) in code generation, focusing on benchmarking their performance across various coding tasks. Leveraging advanced Natural Language Processing (NLP) techniques and deep learning architectures, our study investigates how LLMs, such as the codellama-13b-instruct.Q5_K_S.gguf engine, interpret and generate code from natural language instructions. With an emphasis on accuracy, efficiency, and user accessibility, our research seeks to shed light on the capabilities of LLMs in bridging the gap between human language and executable code. By evaluating factors such as model architecture, training data quality, and task complexity, we aim to provide insights into the potential of LLMs for revolutionizing the coding experience. Through meticulous benchmarking and analysis, this study aims to contribute to the advancement of LLM development and its applications in code generation, paving the way for more efficient and inclusive coding practices in the future.

show abstract

Benchmarking Large Language Models for Automated Verilog RTL Code Generation

Cited by 39 publications

References 3 publications

LADAC: Large Language Model-driven Auto-Designer for Analog Circuits

LADAC: Large Language Model-driven Auto-Designer for Analog Circuits

A survey of safety and trustworthiness of large language models through the lens of verification and validation

Benchmarking Large Language Models for Code Generation

Contact Info

Product

Resources

About