Obtaining ground-truth software architectures

Abstract

Undocumented evolution of a software system and its underlying architecture drives the need for the architecture's recovery from the system's implementation-level artifacts. While a number of recovery techniques have been proposed, they suffer from known inaccuracies. Furthermore, these techniques are difficult to evaluate due to a lack of "ground-truth" architectures that are known to be accurate. To address this problem, we argue for establishing a suite of ground-truth architectures, using a recovery framework proposed in our recent work. This framework considers domain-, application-, and contextspecific information about a system, and addresses an inherent obstacle in establishing a ground-truth architecture -the limited availability of engineers who are closely familiar with the system in question. In this paper, we present our experience in recovering the ground-truth architectures of four open-source systems. We discuss the primary insights gained in the process, analyze the characteristics of the obtained ground-truth architectures, and reflect on the involvement of the systems' engineers in a limited but critical fashion. Our findings suggest the practical feasibility of obtaining ground-truth architectures for large systems and encourage future efforts directed at establishing a large scale repository of such architectures.

A comparative analysis of software architecture recovery techniques

“…In some cases, researchers have used directory or package structure as a widely available architectural proxy. However, our recent study [19] has shown that in most real systems, architects do not consider the directory or package structure to be an accurate reflection of a system's architecture. Finally, the above studies give little guidance as to the conditions under which a recovery technique excels or falters.…”

“…The systems' eight architectures comprise our ground truth. Four of the architectures-those of ArchStudio, Bash, Hadoop, and OODT-were verified as correct by one or more architects of the corresponding systems [19]; the remaining four architectures-two each of Linux and Mozilla-are a result of a meticulous recovery process conducted by other researchers and were used in previous evaluations of recovery techniques [6]. Our assessment is performed at the system-wide level and at the level of individual system components.…”

“…This work informed and aided us in the development of a more recent approach [20] for obtaining the architecture of a software system that can be relied on as its ground truth. We applied that approach in obtaining four of the ground-truth architectures [19] that we use to evaluate the recovery techniques in this paper. Given the limited or no current availability of several recovery techniques' implementations, we reimplemented three of the six techniques used in this study in consultation with their authors.…”

Adaptive Weighting of Structural Dependency and Textual Similarity in Software Architecture Recovery

“…Garcia et al [12] proposed a systematic process for recovering the module view of software manually and published the module views of Apache Hadoop, Apache OODT, and ArchStudio. We shall use them as ground-truth results in our experiments.…”

“…Nowadays various clustering methods and software dependencies for module view recovery are studied in the field of software reengineering. Due to its usefulness, the research on this topic has been actively pursued [12].…”