TANDEM: A Taxonomy and a Dataset of Real-World Performance Bugs

Abstract

The detection of performance bugs, like those causing an unexpected execution time, has gained much attention in the last years due to their potential impact in safety-critical and resourceconstrained applications. Much effort has been put on trying to understand the nature of performance bugs in different domains as a starting point for the development of effective testing techniques. However, the lack of a widely accepted classification scheme of performance faults and, more importantly, the lack of well-documented and understandable datasets makes it difficult to draw rigorous and verifiable conclusions widely accepted by the community. In this paper, we present TANDEM, a dual contribution related to real-world performance bugs. Firstly, we propose a taxonomy of performance bugs based on a thorough systematic review of the related literature, divided into three main categories: effects, causes and contexts of bugs. Secondly, we provide a complete collection of fully documented real-world performance bugs. Together, these contributions pave the way for the development of stronger and reproducible research results on performance testing. INDEX TERMS performance bugs, performance testing, dataset, taxonomy.

SlowCoach: Mutating Code to Simulate Performance Bugs

“…Chen et al [16] surveyed and semantically categorized more than 700 performance bugs from real-world developer commits from 13 popular C/C++ projects. Sánchez et al [17] investigated the performance bugs across multiple publications in the research community and Tizpaz-Niari et al [18] surveyed performance bugs in machine learning libraries.…”

Software Ticks Need No Specifications

“…What we can take away from this work is that: 1) some behaviours are undesirable but need not be defects, 2) some behaviours are trade-offs between one community and another, 3) program properties may be undesirable for developers but invisible to users, and 4) performance issues (including energy, execution time, and memory usage [33]) can be undesirable.…”

“…These two categories can intersect if software quality is part of a program's specification. While practitioners and the literature do not agree on where to draw the line between bug and nonbug issues [24], the nomenclature of performance bugs [33] as well as the popularity of bug patterns for bad coding practices (Section II-B) are points in favour of treating ticks as a category of bugs.…”

Unicorn: Reasoning about Configurable System Performance through the lens of Causality

“…These non-functional faults-unlike regular software bugs-do not cause the system to crash or exhibit any obvious misbehavior [76,85,99]. Instead, misconfigured systems remain operational but degrade in performance [16,71,75,86]. Misconfigurations caused major issues in cloud infrastructure [19], internetscale systems [14], and on-device machine learning systems [3].…”