On the relationship between similar requirements and similar software

Abstract

Recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a stakeholder proposes a new requirement, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn, identify previously developed code. Several NLP approaches for similarity computation between requirements are available. However, there is little empirical evidence on their effectiveness for code retrieval. This study compares different NLP approaches, from lexical ones to semantic, deep-learning techniques, and correlates the similarity among requirements with the similarity of their associated software. The evaluation is conducted on real-world requirements from two industrial projects from a railway company. Specifically, the most similar pairs of requirements across two industrial projects are automatically identified using six language models. Then, the trace links between requirements and software are used to identify the software pairs associated with each requirements pair. The software similarity between pairs is then automatically computed with JPLag. Finally, the correlation between requirements similarity and software similarity is evaluated to see which language model shows the highest correlation and is thus more appropriate for code retrieval. In addition, we perform a focus group with members of the company to collect qualitative data. Results show a moderately positive correlation between requirements similarity and software similarity, with the pre-trained deep learning-based BERT language model with preprocessing outperforming the other models. Practitioners confirm that requirements similarity is generally regarded as a proxy for software similarity. However, they also highlight that additional aspect comes into play when deciding software reuse, e.g., domain/project knowledge, information coming from test cases, and trace links. Our work is among the first ones to explore the relationship between requirements and software similarity from a quantitative and qualitative standpoint. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and change impact analysis.

Making Sense of Failure Logs in an Industrial DevOps Environment

“…A focus group study was planned to collect qualitative insights about the results, following the guidelines by Breen et al [9] and supplemented it with our experience in conducting and reporting focus groups in industry [1][2][3] (see the bottom half of Figure 2). An instrument was drafted and subsequently validated and refined in a pilot session with one expert from the company.…”

“…The repeated execution of regression test suites on different versions and variants of these systems produces a huge number of test execution logs that must be reviewed by engineers [17] [14]. As an example, our industrial partner Westermo Network Technologies AB (Westermo) develops the Westermo Operating System (WeOS) 1 for different industrial switches and routers. WeOS contains the Linux kernel, other free software, as well as proprietary code.…”

Assessing Similarity between Software Requirements: A Semantic Approach

“…Abbas [61] has conducted an empirical investigation of the relationship between requirements and code similarity in the context of a large railway company. The purpose of this work was to explore to what extent similar requirements can be used to locate similar code.…”

“…The purpose of this work was to explore to what extent similar requirements can be used to locate similar code. Abbas [61] analyzed two related projects in the company and used different seminal NLP-based language models to represent the requirements and calculate similarity across the two projects.…”

Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods

“…An example of usage of the NGT technique is presented by Harvey et al [167]. Focus groups are more frequently used in software engineering, typically to involve industrial participants in the validation of prototypical solutions, cf., e.g., Abbas et al [168]. An example of a focus group study in software engineering, carried out via online tools, is presented by Martakis and Daneva [169].…”