Statistical debugging for real-world performance problems

Abstract

Design and implementation defects that lead to inefficient computation widely exist in software. These defects are difficult to avoid and discover. They lead to severe performance degradation and energy waste during production runs, and are becoming increasingly critical with the meager increase of single-core hardware performance and the increasing concerns about energy constraints. Effective tools that diagnose performance problems and point out the inefficiency root cause is sorely needed.The state of the art of performance diagnosis is preliminary. Profiling can tell where computation resources are spent, but not where and why the resources are wasted. Performance-bug detectors can identify specific type of inefficient computation, but are not suited for diagnosing general performance problems. Effective failure diagnosis techniques, such as statistical debugging, have been proposed for functional bugs. However, whether they work for performance problems is still an open question.In this paper, we first conduct an empirical study to understand how performance problems are observed and reported by real-world users. Our study shows that statistical debugging is a natural fit for diagnosing performance problems, which are often observed through comparison-based approaches and reported together with both good and bad inputs. We then thoroughly investigate different design points in statistical debugging, including three different predicates and two different types of statistical models, to understand which design point works the best for performance diagnosis. Finally, we study how some unique nature of performance bugs allows sampling techniques to lower the overhead of run-time performance diagnosis without extending the diagnosis latency.

Static detection of asymptotic performance bugs in collection traversals

“…Song and Lu use five open-source applications to study the use of statistical debugging for finding performance bugs [36]. They find that two kinds of statistical models involving branch predicates can help pinpoint root causes of performance problems.…”

A systematic mapping study of software performance research

“…4 Most existing performance debugging techniques assume that only faulty code could lead to performance bugs. 63 To facilitate performance debugging, practitioners need tools to link the erroneous behavior beyond the source code as the workload level and user configurations could also be the culprit for performance bugs. 9 Researchers need to investigate novel performance debugging techniques to pinpoint the root cause of the more complicated performance bugs.…”

“…While existing debugging techniques 61,62 work well on functional bugs, real‐world performance bugs are much more challenging to handle 4 . Most existing performance debugging techniques assume that only faulty code could lead to performance bugs 63 . To facilitate performance debugging, practitioners need tools to link the erroneous behavior beyond the source code as the workload level and user configurations could also be the culprit for performance bugs 9 .…”

Automated Detection of Software Performance Antipatterns in Java-Based Applications

“…As a motivating example, let us consider a real-life case study [24] where a performance overhead of 17% is experienced for the continuous integration of a software release. The diagnosis of performance problems is indeed non-trivial, a study in [25] indicates the Apache project, i.e., relevant to our research since Java is used, with the longest average diagnosis time (194 days). Our goal is to improve the system performance by identifying the bad practices of software components and fixing them before the system becomes unusable.…”