Statistical debugging for real-world performance problems

Abstract

Design and implementation defects that lead to inefficient computation widely exist in software. These defects are difficult to avoid and discover. They lead to severe performance degradation and energy waste during production runs, and are becoming increasingly critical with the meager increase of single-core hardware performance and the increasing concerns about energy constraints. Effective tools that diagnose performance problems and point out the inefficiency root cause are sorely needed.The state of the art of performance diagnosis is preliminary. Profiling can identify the functions that consume the most computation resources, but can neither identify the ones that waste the most resources nor explain why. Performance-bug detectors can identify specific type of inefficient computation, but are not suited for diagnosing general performance problems. Effective failure diagnosis techniques, such as statistical debugging, have been proposed for functional bugs. However, whether they work for performance problems is still an open question.In this paper, we first conduct an empirical study to understand how performance problems are observed and reported by real-world users. Our study shows that statistical debugging is a natural fit for diagnosing performance problems, which are often observed through comparison-based approaches and reported together with both good and bad inputs. We then thoroughly investigate different design points in statistical debugging, including three different predicates and two different types of statistical models, to understand which design point works the best for performance diagnosis. Finally, we study how some unique nature of performance bugs allows sampling techniques to lower the over- * Shan is now with University of Chicago.

Data-Driven Debugging for Functional Side Channels

“…Statistical Learning for program performance analysis. Machine learning techniques have been used for detecting and explaining performance bugs in software [11], [12], [51]. Tizpaz-Niari et al [12] consider performance issues in softwares.…”

Memory and resource leak defects and their repairs in Java projects

“…To asses the time to resolve (TTR) of an issue report, we adopted the methodology used in previous studies (Song and Lu 2014;Nistor et al 2013;Jin et al 2012b). We collect two timestamps from each issue report: the time it was created (recorded in the issue tracker), and the time it was resolved (labeled as resolved).…”

“…Empirical study. There are many previous works studying characteristics of bugs (Zhong and Su 2015;Jin et al 2012b;Song and Lu 2014;Selakovic and Pradel 2016). Zhong and Su (2015) extracted and analyzed the characteristics of the real bug fixes from six Java projects.…”

Inferring Performance Bug Patterns from Developer Commits

“…A number of articles on approaches to detect or localize performance bugs use actual performance bugs or performance bug simulations to demonstrate the aproaches' efficacy [15]- [17]. These evaluations commonly focus on few bugs (between 15 and 70 in the cited articles) and do not report the relative occurrence of the targeted bug types compared to other bug types or the complexity of the targeted bugs.…”