Validating a model-driven software architecture evaluation and improvement method: A family of experiments

Abstract

ElsevierGonzález Huerta, J.; Insfrán Pelozo, CE.; Abrahao Gonzales, SM.; Scanniello, G. (2015 Context: Software architectures should be evaluated during the early stages of software development in order to verify whether the Non-Functional Requirements (NFRs) of the product can be fulfilled. This activity is even more crucial in Software Product Line (SPL) development, since it is also necessary to identify whether the NFRs of a particular product can be achieved by exercising the variation mechanisms provided by the product line architecture or whether additional transformations are required. These issues have motivated us to propose QuaDAI, a method for the derivation, evaluation and improvement of software architectures in model-driven SPL development.Objective: We present in this paper the results of a family of four experiments carried out to empirically validate the evaluation and improvement strategy of QuaDAI. Method:The family of experiments was carried out by 92 participants: Computer Science Master's and undergraduate students from Spain and Italy. The goal was to compare the effectiveness, efficiency, perceived ease of use, perceived usefulness and intention to use with regard to participants using the evaluation and improvement strategy of QuaDAI as opposed to the Architecture Tradeoff Analysis Method (ATAM). Results:The main result was that the participants produced their best results when applying QuaDAI, signifying that the participants obtained architectures with better values for the NFRs faster, and that they found the method easier to use, more useful and more likely to be used. The results of the meta-analysis carried out to aggregate the results obtained in the individual experiments also confirmed these results. Conclusions:The results support the hypothesis that QuaDAI would achieve better results than ATAM in the experiments and that QuaDAI can be considered as a promising approach with which to perform architectural evaluations that occur after the product architecture derivation in model-driven SPL development processes when carried out by novice software evaluators.

Enhanced Flexibility and Reusability through State Machine-Based Architectures for Multisensor Intelligent Robotics

“…In the last few years, several methods for evaluating software architectures have been defined: scenario-based (SAAM, architecture tradeoff analysis method (ATAM), ALMA , etc.) [ 40 , 41 , 42 , 43 , 44 , 45 ], mathematical model-based (reliability analysis, performance analysis) [ 46 ] and metrics-based software architecture evaluation methods (QuADAI ) [ 47 ]. In order to evaluate the advantages of the proposed architecture, based on the previously-mentioned methods, a simplified approach of the architecture tradeoff analysis method (ATAM) has been selected to perform a comparative analysis [ 40 , 48 , 49 ].…”

Enhanced Flexibility and Reusability Through State Machine Based Architectures for Multisensor Intelligent Robotics

“…[40][41][42][43], mathematical model-based (Reliability Analysis, Performance Analysis) [44] and metrics-based software architecture evaluation methods (QuADAI) [45]. In order to evaluate the advantages of the proposed architecture, based on previously mentioned methods, a simplified approach of Architecture Tradeoff Analysis Method (ATAM) method has been selected to perform a comparative analysis [40,[46][47][48].…”

Variability Resolution and Product Configuration with SMarty: An Experimental Study on UML Class Diagrams

“…Besides the results of the systematic review two studies realized experimental evaluations in evaluation of architecture of software product line Gonzalez-Huerta et al (2015) and for a UML-based approach to modeling variabilities, such as SMarty and PLUS (Reinhartz-Berger and Sturm, 2014). Gonzalez-Huerta et al (2015), after an analysis of literature identified a low number of empirical validations of evaluation methods for software architectures.…”

“…Gonzalez-Huerta et al (2015), after an analysis of literature identified a low number of empirical validations of evaluation methods for software architectures. Such a low number of empirical studies, quantitative and qualitative comparisons with existing methods has been neglected and the low number of studies replications.…”

“…Participants measure an architecture to check the fulfillment of the non-functional requirements with the two methods. Such as our study, which is part of a family of experimental studies, Gonzalez-Huerta et al (2015) presented a new study and their family of experimental studies consolidate in a meta-analysis. The experimental analysis indicated that participants produced their best results when applying QuaDAI and the meta-analysis aggregate the results obtained in the individual experiments.…”