Realizing self-adaptive systems via online reinforcement learning and feature-model-guided exploration

Abstract

A self-adaptive system can automatically maintain its quality requirements in the presence of dynamic environment changes. Developing a self-adaptive system may be difficult due to design time uncertainty; e.g., anticipating all potential environment changes at design time is in most cases infeasible. To realize self-adaptive systems in the presence of design time uncertainty, online machine learning, i.e., machine learning at runtime, is increasingly used. In particular, online reinforcement learning is proposed, which learns suitable adaptation actions through interactions with the environment at runtime. To learn about its environment, online reinforcement learning has to select actions that were not selected before, which is known as exploration. How exploration happens impacts the performance of the learning process. We focus on two problems related to how adaptation actions are explored. First, existing solutions randomly explore adaptation actions and thus may exhibit slow learning if there are many possible adaptation actions. Second, they are unaware of system evolution, and thus may explore new adaptation actions introduced during evolution rather late. We propose novel exploration strategies that use feature models (from software product line engineering) to guide exploration in the presence of many adaptation actions and system evolution. Experimental results for two realistic self-adaptive systems indicate an average speed-up of the learning process of 33.7% in the presence of many adaptation actions, and of 50.6% in the presence of evolution.

Explaining Online Reinforcement Learning Decisions of Self-Adaptive Systems

“…The left hand side of Fig. 4 shows the contents of the main part of the XRL-DINE dashboard for the example 3 . This figure shows an excerpt of the overall learning process.…”

“…1(c). For a self-adaptive system, "agent" refers to the selfadaptation logic of the system and "action" refers to an adaptation action [3]. In the integrated model, action selection of RL takes the place of the analyze and plan activities of MAPE-K.…”

“…One key element of a self-adaptive system is its self-adaptation logic that encodes when and how the system should adapt itself. When developing the adaptation logic, developers face the challenge of design time uncertainty [2], [3], [4]. To define when the system should adapt, they have to anticipate all potential environment states.…”

“…Online reinforcement learning (Online RL) is an emerging approach to realize self-adaptive systems in the presence of design time uncertainty. Online RL means that reinforcement learning [5] is employed at runtime (see [3] for a discussion of existing solutions). The self-adaptive system thereby can learn from actual operational data and thus leverages information only available at runtime.…”

Configuration Optimization with Limited Functional Impact

“…Based on this observation and inspired by our previous work on measuring the energy consumption of configurable systems [2], we propose in this paper an approach that optimizes a configuration regarding multiple performance objectives. Contrarily to prior work that samples or predicts performance models seeking for the best configuration of the whole configuration space [4,7,9,10,22], our approach optimizes existing configurations by maximizing performance gains while minimizing changes to such configurations. The objective is to provide the developer with the best-performing configuration by altering as little as possible the initial one, in order to remain as close as possible to the developer's functional requirements.…”