Source Code Summarization with Structural Relative Position Guided Transformer

Gong, Zi; Gao, Cuiyun; Wang, Yasheng; Gu, Wenchao; Peng, Yun; Xu, Zenglin

doi:10.1109/saner53432.2022.00013

Cited by 20 publications

(13 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SCRIPT [ 11 ] introducing the structural relative positions between nodes of the AST to better capture the structural relative dependencies.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…Ahmad et al [ 8 ] first proposed a transformer-based method on the code summarization task, which achieved excellent performance and leads the code summarization area into the transformer-based model stage. Because of the popularization and performance of transformers, almost all recent works [ 9 , 10 , 11 , 12 ] are conducted based on the transformer architecture and achieve high scores in each evaluation metric. However, only considering sequence information without considering the structure of code leads to a incomplete representation of code.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Structure and Sequence Aligned Code Summarization with Prefix and Suffix Balanced Strategy

Zeng

Cai

2023

Entropy

View full text Add to dashboard Cite

Source code summarization focuses on generating qualified natural language descriptions of a code snippet (e.g., functionality, usage and version). In an actual development environment, descriptions of the code are missing or not consistent with the code due to human factors, which makes it difficult for developers to comprehend and conduct subsequent maintenance. Some existing methods generate summaries from the sequence information of code without considering the structural information. Recently, researchers have adopted the Graph Neural Networks (GNNs) to capture the structural information with modified Abstract Syntax Trees (ASTs) to comprehensively represent a source code, but the alignment method of the two information encoder is hard to decide. In this paper, we propose a source code summarization model named SSCS, a unified transformer-based encoder–decoder architecture, for capturing structural and sequence information. SSCS is designed upon a structure-induced transformer with three main novel improvements. SSCS captures the structural information in a multi-scale aspect with an adapted fusion strategy and adopts a hierarchical encoding strategy to capture the textual information from the perspective of the document. Moreover, SSCS utilizes a bidirectional decoder which generates a summary from opposite direction to balance the generation performance between prefix and suffix. We conduct experiments on two public Java and Python datasets to evaluate our method and the result show that SSCS outperforms the state-of-art code summarization methods.

show abstract

“…SCRIPT [ 11 ] introducing the structural relative positions between nodes of the AST to better capture the structural relative dependencies.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Structure and Sequence Aligned Code Summarization with Prefix and Suffix Balanced Strategy

Zeng

Cai

2023

Entropy

View full text Add to dashboard Cite

show abstract

“…LLMs (e.g., GPT-3 [26], CodeX [14], ChatGPT [42]), have shown significant improvements in software engineering tasks, such as requirements classification [43], [44], [45], FQN inference [46], [47], and code summarization [48], [49], [50]. They can capture code's structural knowledge (e.g., AST [51], [52]) and semantic knowledge (e.g., code weakness [53], [54] and API relation [23]).…”

Section: Related Workmentioning

confidence: 99%

Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code

Huang

Yuan

Xing

et al. 2022

Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Control Flow Graphs (CFGs) are essential for visualizing, understanding and analyzing program behavior. For statically-typed programming language like Java, developers obtain CFGs by using bytecode-based methods for compilable code and Abstract Syntax Tree (AST)-based methods for partially uncompilable code. However, explicit syntax errors during AST construction and implicit semantic errors caused by bad coding practices can lead to behavioral loss and deviation of CFGs. To address the issue, we propose a novel approach that leverages the error-tolerant and understanding ability of pre-trained Large Language Models (LLMs) to generate CFGs. Our approach involves a Chain of Thought (CoT) with four steps: structure hierarchy extraction, nested code block extraction, CFG generation of nested code blocks, and fusion of all nested code blocks' CFGs. To address the limitations of the original CoT's single-prompt approach (i.e., completing all steps in a single generative pass), which can result in an "epic" prompt with hard-to-control behavior and error accumulation, we break down the CoT into an AI chain with explicit sub-steps. Each sub-step corresponds to a separate AI-unit, with an effective prompt assigned to each unit for interacting with LLMs to accomplish a specific purpose. Our experiments confirmed that our method outperforms existing CFG tools in terms of node and edge coverage, especially for incomplete or erroneous code. We also conducted an ablation experiment and confirmed the effectiveness of AI chain design principles: Hierarchical Task Breakdown, Unit Composition, and Mix of AI Units and Non-AI Units. Our work opens up new possibilities for building foundational software engineering tools based on LLMs, as opposed to traditional program analysis methods.

show abstract

“…It is well known that incorporating structural inductive priors -which is usually implemented via various additive relative masking mechanisms in regular attention architectures -is difficult for Performers. We refer to these methods as Relative Positional Encodings (or RPEs) (Shaw et al, 2018;Raffel et al, 2020;Wu et al, 2021;Gong et al, 2022;Luo et al, 2022b). RPEs play a critical role in improving the performance of Transformers in long-range modeling, e.g.…”

Section: Introductionmentioning

confidence: 99%

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Choromański¹,

Li²,

Likhosherstov³

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for nongeometric data, as well as novel RPEs operating on the sequences of tokens embedded in higherdimensional Euclidean spaces (e.g. point clouds). FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE-mask. FLTs allow also for applying certain structural inductive bias techniques to specify masking strategies, e.g. they provide a way to learn the so-called local RPEs introduced in this paper and providing accuracy gains as compared with several other linear Transformers for language modeling. We also thoroughly tested FLTs on other data modalities and tasks, such as: image classification and 3D molecular modeling. For 3D-data FLTs are, to the best of our knowledge, the first Transformers architectures providing RPE-enhanced linear attention.

show abstract

Source Code Summarization with Structural Relative Position Guided Transformer

Cited by 20 publications

References 38 publications

Structure and Sequence Aligned Code Summarization with Prefix and Suffix Balanced Strategy

Structure and Sequence Aligned Code Summarization with Prefix and Suffix Balanced Strategy

Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Contact Info

Product

Resources

About