CoTexT: Multi-task Learning with Code-Text Transformer

Phan, Long; Tran, Hieu; Le, Daniel; Nguyen, Hieu; Anibal, James; Peltekian, Alec; Ye, Yanfang

doi:10.48550/arxiv.2105.08645

Cited by 9 publications

(17 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• RQ3: How effective is CDCS applied on other pre-trained programming language models? Besides CodeBERT, there are other pre-trained models that also achieve outstanding results in software engineering tasks [1,22,24]. We wonder whether other pre-trained models can have the same effectiveness on code search when equipped with meta learning.…”

Section: Methodsmentioning

confidence: 99%

“…In recent years, pre-trained language models for source code have received much attention [1,10,22,24]. CodeBERT [10], built on top of the popular model of BERT [8], is one of the earliest attempts that adapt pre-trained models for programming languages.…”

Section: Pre-trained Language Models For Codementioning

confidence: 99%

“…Besides, they creatively proposed the replaced token detection (RTD) task for the pre-training of programming language. CoText [24] is a pre-trained Transformer model for both natural language and programming languages. It follows the encoder-decoder architecture proposed by [32].…”

Section: Pre-trained Language Models For Codementioning

confidence: 99%

See 2 more Smart Citations

Cross-Domain Deep Code Search with Few-Shot Meta Learning

Chai,

Zhang,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, pre-trained programming language models such as Code-BERT have demonstrated substantial gains in code search. Despite their success, they rely on the availability of large amounts of parallel data to fine-tune the semantic mappings between queries and code. This restricts their practicality in domain-specific languages with relatively scarce and expensive data. In this paper, we propose CDCS, a novel approach for domain-specific code search. CDCS employs a transfer learning framework where an initial program representation model is pre-trained on a large corpus of common programming languages (such as Java and Python), and is further adapted to domain-specific languages such as SQL and Solidity. Unlike cross-language CodeBERT, which is directly fine-tuned in the target language, CDCS adapts a few-shot meta-learning algorithm called MAML to learn the good initialization of model parameters, which can be best reused in a domain-specific language. We evaluate the proposed approach on two domain-specific languages, namely, SQL and Solidity, with model transferred from two widely used languages (Python and Java). Experimental results show that CDCS significantly outperforms conventional pre-trained code models that are directly fine-tuned in domain-specific languages, and it is particularly effective for scarce data. CCS CONCEPTS• Software and its engineering → Reusability; Automatic programming.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Pre-trained Language Models For Codementioning

confidence: 99%

See 1 more Smart Citation

Cross-Domain Deep Code Search with Few-Shot Meta Learning

Chai,

Zhang,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…More recent approaches to code documentation utilize pretrained transformers. Currently, CoTexT (Phan et al, 2021) outperforms CodeBERT , PLBART , and ProphetNET-X (Qi et al, 2021) in this task. The CodeSearchNet corpus is further described in §3.4 and the models are described later on in the code generation subsection §3.8.…”

Section: Code Documentation Generation and Summarizationmentioning

confidence: 99%

“…Other pretrained transformers used on source code include CodeT5 (Wang et al, 2021b). Code-Trans (Elnaggar et al, 2021), PyMT5 (Clement et al, 2020), CuBERT (Kanade et al, 2020), PLBART , ProphetNet-X (Qi et al, 2021), CoTexT (Phan et al, 2021), T5-Code (Mastropaolo et al, 2021), GraphCode-BERT , and AlphaCode (Li et al, 2022). Pretrained GPT-style Models for source code generation include CodeGPT , and GPT-Codex (Chen et al, 2021a).…”

Section: Pretrained Transformer Modelsmentioning

confidence: 99%

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

Al-Hossami¹,

Shaikh²

2022

Preprint

View full text Add to dashboard Cite

In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also known as Code Intelligence (CI) and Programming Language Processing (PLP). We survey over 287 publications and present a software-engineering centered taxonomy for CI placing each of the works into one category describing how it best assists the software development cycle. Then, we overview the field of conversational assistants and their applications in software engineering and education. Lastly, we highlight research opportunities at the intersection of AI for code and conversational assistants and provide future directions for researching conversational assistants with CI capabilities.

show abstract

An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering

Li,

Shi,

Zhang

2024

IEEE Access

View full text Add to dashboard Cite

Code generation stands as a powerful technique in modern software development, improving development efficiency, reducing errors, and fostering standardization and consistency. Recently, ChatGPT has exhibited immense potential in automatic code generation. However, existing researches on code generation lack guidance for practical software development process. In this study, we utilized ChatGPT to develop a web-based code generation platform consisting of key components: User Interface, Prompt Builder, and Backend Service. Specifically, Prompt Builder dynamically generated comprehensive prompts to enhance model generation performance. We conducted experiments on 2 datasets to evaluate the performance of code generation in our approach. through 8 widely used metrics. The results demonstrate that (1) our Prompt Builder is effective, resulting in a 65.06% improvement in the exact match (EM), a 38.45% improvement in Bilingual Evaluation Understudy (BLEU), a 15.70% improvement in CodeBLEU, and a 50.64% improvement in Pass@1. (2) In real development scenarios, 98.5% of test cases can be validated through manual validation, highlighting the genuine assistance provided by the ChatGPT-based code generation approach.

show abstract

CoTexT: Multi-task Learning with Code-Text Transformer

Cited by 9 publications

References 11 publications

Cross-Domain Deep Code Search with Few-Shot Meta Learning

Cross-Domain Deep Code Search with Few-Shot Meta Learning

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering

Contact Info

Product

Resources

About