Traceability Transformed: Generating More Accurate Links with Pre-Trained BERT Models

Abstract

Software traceability establishes and leverages associations between diverse development artifacts. Researchers have proposed the use of deep learning trace models to link natural language artifacts, such as requirements and issue descriptions, to source code; however, their effectiveness has been restricted by availability of labeled data and efficiency at runtime. In this study, we propose a novel framework called TVace BERT (T-BERT) to generate trace links between source code and natural language artifacts. To address data sparsity, we leverage a three-step training strategy to enable trace models to transfer knowledge from a closely related Software Engineering challenge, which has a rich dataset, to produce trace links with much higher accuracy than has previously been achieved. We then apply the T-BERT framework to recover links between issues and commits in Open Source Projects. We comparatively evaluated accuracy and efficiency of three BERT architectures. Results show that a Single-BERT architecture generated the most accurate links, while a Siamese-BERT architecture produced comparable results with significantly less execution time. Furthermore, by learning and transferring knowledge, all three models in the framework outperform classical IR trace models. On the three evaluated real-word OSS projects, the best T-BERT stably outperformed the VSM model with average improvements of 60.31% measured using Mean Average Precision (MAP). RNN severely underperformed on these projects due to insufficient training data, while T-BERT overcame this problem by using pretrained language models and transfer learning.

Making the most of small Software Engineering datasets with modern machine learning

“…Another alternative is to define suitable intermediate training tasks. We have found initial evidence of this, and a recent paper adds further evidence, in the context of traceability [83]. However, it used a very similar task and dataset.…”

Improving Code Autocompletion with Transfer Learning

“…Outside of code authoring, transfer learning has been enabled by Transformer-based models in other software artifacts. Lin et al [24] apply pretrained BERT models for learning relationships between issues and commits in a software repos-itory. Sharma et al [25] detect code smells in programming languages where sufficient training data is not available by transferring knowledge from other data-rich languages.…”

PTM4Tag: Sharpening Tag Recommendation of Stack Overflow Posts with Pre-trained Models

“…In the future, we plan to consider more diverse information of a post into account, such as the attached pictures, author information, etc. Also, we are interested in applying PTM4Tag on more SQA sites such as Freecode 14 , AskUbuntu 15 , etc., to further evaluate its effectiveness and generalizability. We release our replication package 16 to facilitate future research.…”

“…Recent trends in the NLP domain have led to the rapid development of transfer learning. Especially, substantial work has shown that pre-trained language models learn practical and generic language representations which could achieve outstanding performance in various downstream tasks simply by fine-tuning, i.e., without training a new model from scratch [10,15,20]. With proper training manner, the model can effectively capture the semantics of individual words based on their surrounding context and reflect the meaning of the whole sentence.…”