Evaluation of Textual Similarity Techniques in Code Level Traceability

Abstract

Automatic recovery of test-to-code traceability links is an important task in many areas of software engineering, like quality assurance and code maintenance. The research community has shown great interest in such a topic and has developed several techniques that already made significant advances in the field. These techniques include text-based learning algorithms, of which corpus is built from the source code of the software components. Several techniques based on information retrieval have been benchmarked, but the capabilities of many learning algorithms have not yet been tested. In this work we examine the textual similarity measures produced by three different machine learning techniques for the recovery of traceability information while also considering various textual representations of the source code. The obtained results are evaluated on 4 open source systems based on naming conventions. We have been able to improve the current textual similarity based state-of-the-art results in the case of each evaluated system.

Large Scale Evaluation of Natural Language Processing Based Test-to-Code Traceability Approaches

“…Thus, finding better performing textual methods can improve these possible combinations as well, having the potential of major contributions to the field. Our findings [4] show that improved versions of lexical analysis can significantly outshine the previously mentioned low results, raising their average precision over 50%.…”

“…Thus, our results were evaluated on eight real opensource programs and also using a variety of source code representations and settings. Our previous work aimed to show that LSI itself performs better than it was previously perceived by the research community [5], investigated the question of source code representations in the task, and also found that Doc2Vec can significantly outperform LSI [6] while a suitable combination of the textual similarity techniques could provide even better results [4]. Some of the approaches used in the paper are also defined in our previous work but they are also briefly introduced in the following subsections.…”

Utilizing Source Code Embeddings to Identify Correct Patches

“…For each program in the database, we obtained three different textual representations to measure similarity. We already described these representations in more detail in our previous works [24], [25]. Similar representations are widely used in other research experiments, such as [2], [26].…”

“…Recommendation systems are not new to software engineering [39]- [41], presenting a prioritized list of most likely solutions seems to be a more resilient approach even in traceability research [24], [25], [42].…”

“…: sentences, documents). Although Doc2vec [49] not enjoys the immense popularity of word2vec, it is still well-known to the scientific community [24], [25], [50]- [53], although they are much less prevalent in the field of software engineering. Similarly to Doc2vec, the use of Bert [17] in this specific domain is not very widespread.…”