Criteria for the evaluation of self-* systems

Abstract

International audienceIn the last years, the growing complexity of the current applications has led to the design of self-adapting systems presenting self-* properties. These systems are composed of several autonomous interactive entities. They behave autonomously and present enhanced characteristics allowing them to handle dynamics coming from exogenous and endogenous changes. In this paper, we propose a set of criteria for the description and evaluation of the adaptive properties of such systems. They aim to provide a concrete mechanism to analyze the quality of the design of adaptive systems, to evaluate the effect of self-* properties on the performances and to compare the adaptive features of different systems. The criteria are grouped into different categories: methodological, architectural, intrinsic, and runtime evaluation. They have been identified and specified by analyzing several case studies, which address self-adaptivity issues through different approaches with different objectives in various application contexts

Empirical resilience evaluation of an architecture-based self-adaptive software system

“…We have complemented our study with additional evidence collected according to some of the run-time evaluation criteria proposed in [20] that indicate the ability of the system to return to its COP after a disturbance:…”

“…The criteria for evaluation of adaptive properties presented in [20] and [30] aim at assessing the impact of self-* properties on different aspects of the system, such as, performance, in addition to comparing the adaptive features of different systems. Concretely, the criteria in [20] are grouped in different categories, among which "run-time evaluation" is the one that has a stronger relation with resilience.…”

“…Concretely, the criteria in [20] are grouped in different categories, among which "run-time evaluation" is the one that has a stronger relation with resilience. The definition of the different criteria relies on concepts such as self-* situations and nominal situation, comparable to NCOPs and COP, respectively (even if they are not formally defined).…”

“…Although this approach is intended to help architects in the generation of adaptable architectures at development-time, the authors propose as future work its integration at run-time, extending it with the metrics presented in [20].…”

A Stepwise Refinement Based Development of Self-Organizing Multi-Agent Systems: Application to the Foraging Ants

“…Some works propose using test and simulation techniques [5], others define metrics for evaluating the resulting behaviour of the system [6]. These techniques offer an experimental way to verify SOMAS but don't give any formal guarantee.…”