Performance problem diagnostics by systematic experimentation

Abstract

The performance of enterprise software systems has a direct impact on the success of business. Recent studies have shown that software performance affects customer satisfaction as well as operational costs of software. Hence, software performance constitutes an essential competitive and differentiating factor for software vendors and operators. In industrial practice, it is still a challenging task to detect software performance problems before they are faced by end users. Diagnostics of performance problems requires deep expertise in performance engineering and still entails a high manual effort. As a consequence, performance evaluations are postponed to the last minute of the development process, or even are completely omitted. Instead of proactively avoiding performance problems, problems are fixed in a reactive manner when they first emerge in operations. Since reactive, operation-time resolution of performance problems is very expensive and has a negative impact on the reputation of software vendors, performance problems need to be diagnosed and resolved in the process of software development. Existing approaches addressing performance problem diagnostics either assume the existence of a performance model, are limited to problem detection without analyzing root causes, or are applied as reactive approaches during the operations phase and, thus, cannot be effective applied during development for performance problem diagnostics.In this thesis, we introduce an automatic, experiment-based approach for performance problem diagnostics in enterprise software systems. We describe a method to derive a taxonomy on recurrent types of performance problems and introduce a systematic experimentation concept. Using the taxonomy as a search tree, the proposed approach systematically searches for root causes of detected performance problems by executing series of systematic performance experiments. Based on the measurement data from experiments, detection heuristics decide on the presence of performance problems in the target system. Furthermore, we develop a domain-specific description language to specify the information required for automatic performance problem diagnostics. Finally, we create and evaluate a representative set of detection heuristics. We validate our approach by means of five studies including end-to-end case studies, a controlled experiment and an empirical study. The results of the validation show that our approach is applicable to a wide range of contexts and is able to fully automatically and accurately detect performance problems in medium-size and large-scale applications. External users of the provided approach evaluated it as a useful support for diagnostics of performance problems and exposed their willingness to use the approach for their own software development projects. Explicitly designed for automatic, development-time testing, our approach can be incorporated into continuous integration. In this way, our approach allows regular, automatic diagnostics of performance problems involving minima...

Detecting and Localizing Anomalies in Container Clusters Using Markov Models

“…Peiris et al [21] analyze the root causes of performance anomalies of CPU usage by combining the correlation and comparative analysis techniques in distributed environments. The work in [11] analyzes workload to detect performance anomalies and categorizes performance anomalies into three layers: (1) symptoms, externally visible indicators of a performance problem, (2) manifestation, internal performance indicators or evidences, and (3) root causes, physical factors whose removal eliminates the manifestations and symptoms of a performance incident.…”

“…There are many studies regarding detecting anomalies in cloud and edge computing contexts [6,7], and specifically in containerized environments [8][9][10]. Some studies looked at detecting an anomaly's root cause in clouds at a virtual machine [7,11] or network level [12]. Detecting and localizing anomalies in clustered container deployments [13] is still recognized as a research gap [8,14].…”

Automatic detection of performance anti-patterns in inter-component communications

“…In previous work [23,24] we made the observation that different performance anti-patterns exhibit similar symptoms, which provides a more systematic way to structure performance anti-patterns. While the focus of the performance problem structure in [24] was on performance bottlenecks, in this paper we extend the hierarchy with CPAs.…”

“…In [23,24] we introduced an extensible framework for automated detection of performance problems (in the following referred to as Dynamic Spotter ) based on systematic experimentation. The CPA detection heuristics presented in this paper are integrated with the Dynamic Spotter framework.…”

A Multivariate Characterization and Detection of Software Performance Antipatterns

“…The extensive body of knowledge has addressed several aspects of Software Performance Antipatterns (SPAs), as for example SPA classification and solution [2], early detection/solution at the design phase [3], methodologies to rank SPAs occurring in design models [4], detection/solution during the testing or operational phases [5], load testing and profiling to detect SPAs in Java applications [6], and an automated approach for detection in load testing and production [7].…”