Performance analysis of idle programs

Abstract

This paper presents an approach for performance analysis of modern enterprise-class server applications. In our experience, performance bottlenecks in these applications differ qualitatively from bottlenecks in smaller, stand-alone systems. Small applications and benchmarks often suffer from CPU-intensive hot spots. In contrast, enterprise-class multi-tier applications often suffer from problems that manifest not as hot spots, but as idle time , indicating a lack of forward motion. Many factors can contribute to undesirable idle time, including locking problems, excessive system-level activities like garbage collection, various resource constraints, and problems driving load. We present the design and methodology for WAIT, a tool to diagnosis the root cause of idle time in server applications. Given lightweight samples of Java activity on a single tier, the tool can often pinpoint the primary bottleneck on a multi-tier system. The methodology centers on an informative abstraction of the states of idleness observed in a running program. This abstraction allows the tool to distinguish, for example, between hold-ups on a database machine, insufficient load, lock contention in application code, and a conventional bottleneck due to a hot method. To compute the abstraction, we present a simple expert system based on an extensible set of declarative rules. WAIT can be deployed on the fly, without modifying or even restarting the application. Many groups in IBM have applied the tool to diagnosis performance problems in commercial systems, and we present a number of examples as case studies.

Statistical debugging for real-world performance problems

“…For C programs, we use CBI [22,23] to collect all these three types of predicates 3 . For C++ programs, we implement our own branch-predicate and return-predicate collection tools using PIN binary-instrumentation framework [27].…”

“…Jin et al [18] employ rule-based methods to detect performance bugs that violate efficiency rules that have been violated before. WAIT [3] focuses on bugs that block the application from making progress. Liu and Berger [24] build two tools to attack the false sharing problem in multi-threaded software.…”

In-Test Adaptation of Workload in Enterprise Application Performance Testing

“…As the application to test, we used JPetStore [5], an open-source e-commerce application, which is commonly used in the literature [17,18]. Also, we used IBM WAIT as diagnosis tool due to its strong analytic capabilities to detect performance issues (e.g., lock contention or database bottleneck) in Java systems [8]. Two types of runs were performed: Following common industry practices [7], the first run type used the traditional approach of static workloads (in the range of [100..2000] concurrent virtual users in increments of 100) and was considered the baseline.…”

Statistical debugging for real-world performance problems

“…Automated failure diagnosis has been studied for decades for functional bugs 3 . Many useful and generic techniques [16,19,20,22,29,50] have been proposed.…”

“…Chen et al [9] detect database related performance anti-patterns, like fetching excessive data from database and issuing queries that could have been aggregated. WAIT [3] focuses on bugs that block the application from making progress. Liu and Berger [31] build two tools to attack the false sharing problem in multithreaded software.…”