The Performance Cockpit Approach: A Framework For Systematic Performance Evaluations

Abstract

Abstract-Evaluating the performance (timing behavior, throughput, and resource utilization) of a software system becomes more and more challenging as today's enterprise applications are built on a large basis of existing software (e.g. middleware, legacy applications, and third party services). As the performance of a system is affected by multiple factors on each layer of the system, performance analysts require detailed knowledge about the system under test and have to deal with a huge number of tools for benchmarking, monitoring, and analyzing. In practice, performance analysts try to handle the complexity by focusing on certain aspects, tools, or technologies. However, these isolated solutions are inefficient due to the small reuse and knowledge sharing. The Performance Cockpit presented in this paper is a framework that encapsulates knowledge about performance engineering, the system under test, and analyses in a single application by providing a flexible, plug-in based architecture. We demonstrate the value of the framework by means of two different case studies.

Collaborative Software Performance Engineering for Enterprise Applications

“…Furthermore, model-based techniques require expert knowledge about the system to be implemented and its dependencies which may not be available especially for third-party products that are used in application system landscapes [5]. Additionally, identifying the parameters of the performance model can be difficult, e.g.…”

“…Hence, measurement-based prediction techniques consider all executional dependencies which model-based approaches cannot guarantee [16]. On the one hand, measurement-based approaches are more used in practice than modelbased approaches due to their effectiveness in operation [5]. On the other hand, they usually allow for fewer degrees of freedom than model-based approaches resulting in a trade-off between runtime and accuracy [11].…”

“…Therefore, DevOps enables a culture, practices and automation that support fast, efficient and reliable software delivery [4]. However, especially for enterprise applications, IT departments usually make use of existing standard software components instead of developing solutions entirely in-house [5]. In such cases, activities referred to as Dev, rather include requirements engineering, architectural design, customization, testing, performance-tuning and deployment [6].…”

Capacity Management as a Service for Enterprise Standard Software

“…Unfortunately, such concepts and methods related to performance modeling and analysis are largely unknown to most IT practitioners [24] and, thus, are rarely applied in industry [25]. Main reasons include a generally high time effort in construction and analysis of analytical models [26] as well as essential expert knowledge about the model itself, the system to be implemented and its dependencies which may not be available [27]. In addition, the credibility of obtained results remains questionable until their validation in later lifecycle phases [6].…”

“…In addition, the credibility of obtained results remains questionable until their validation in later lifecycle phases [6]. Therefore, measurementbased approaches are more frequently used in practice than model-based approaches due to their effectiveness in operation [27]. As a costly consequence, performance requirements are considered lately and performance testing is done when an implemented or relocated system is about to go live [28], [29].…”