Changeloads for Resilience Benchmarking of Self-Adaptive Systems: A Risk-Based Approach

Abstract

Benchmarking self-adaptive software systems calls for a new model that takes into account a distinctive characteristic of such systems: alterations over time (i.e., self-achieved modifications or adjustments triggered by changes in the external or internal contexts of the system). Changes are thus a fundamental component of a resilience benchmark, raising an intrinsic research problem: how to identify and select the most realistic and relevant (sequences of) changes to be included in the benchmarking procedure. The problem is that defining a representative changeload would require access to a large amount of field data, which is not available for most systems. In this paper we propose an approach based on risk analysis to tackle this key issue, debating its effectiveness and usability with a simple case study. The procedure, that combines field data with expert knowledge and experimental data, allows moving from the identification of the generic goals of systems in the benchmarking domain to the identification of the most relevant change scenarios (based on probability and impact) that may prevent those systems from achieving their goals.

SCoRe: An across-the-board metric for computer systems resilience benchmarking

“…), but distinct from system security, which is defined as "absence of unauthorized access to, or handling of, system state". In this perspective, we refined this view of the resilience of a system as its ability to maintain service delivery that can justifiably be trusted in spite of changes in its internal and external contexts [20].…”

“…A noteworthy aspect of this catalogue is that not only does it include major life events (clearly important sources of risk and adversity), but also daily stresses (sources of risk that have lower severity but greater chronicity) [8], to gain a more complete picture of risk and adversity. This approach is, in its essence, similar to the one proposed in [20], where the set of change scenarios that are imposed on the system for resilience benchmarking are defined using a risk analysis approach to select the changes that have the higher impact x probability of occurrence. This approach aims at evaluating the behavior of the system not only when faced with the changes that will most probably have a greater impact on system goals, but also with ones that, by being very frequent, also impose a continuous stress on the system.…”

“…The problem is that accessing such data is not possible for most systems and/or domains, either because changes are not recorded or due to intellectual property issues. This way, to support the definition of real resilience benchmarks, a more practical approach was proposed in [20].…”

Empirical resilience evaluation of an architecture-based self-adaptive software system

“…Compared to established benchmarks, a resilience benchmark is specified following the same basic approach, but comprising a wide-ranging changeload (including, but not limited to, faults) and resilience metrics [1].…”

“…Our main contribution is an empirical study about the impact of architecture-based self-adaptation on resilience, which employs a framework for evaluating resilience in self-adaptive systems [7] incorporating the notion of changeload [1], which includes changes in the system and its environment.…”