Input-sensitive profiling

Abstract

Input-sensitive profiling is a recent performance analysis technique that makes it possible to estimate the empirical cost function of individual routines of a program, helping developers understand how performance scales to larger inputs and pinpoint asymptotic bottlenecks in the code. A current limitation of input-sensitive profilers is that they specifically target sequential computations, ignoring any communication between threads. In this paper we show how to overcome this limitation, extending the range of applicability of the original approach to multithreaded applications and to applications that operate on I/O streams. We develop new metrics for automatically estimating the size of the input given to each routine activation, addressing input produced by nondeterministic memory stores performed by other threads as well as by the OS kernel (e.g., in response to I/O or network operations). We provide real case studies, showing that our extension allows it to characterize the behavior of complex applications more precisely than previous approaches. An extensive experimental investigation on a variety of benchmark suites (including the SPEC OMP2012 and the PARSEC benchmarks) shows that our Valgrind-based input-sensitive profiler incurs an overhead comparable to other prominent heavyweight analysis tools, while collecting significantly more performance points from each profiling session and correctly characterizing both thread-induced and external input.

Performance localisation

“…Profiling is the cheapest analysis to perform as instrumentation only need be performed once and a single execution of a program is needed to gather results. Even if we use profiling to find each statement's sensitivity to input size [3], we may only need use a small number of test cases to find this information. As evaluation time dominates, we use this as our measure of computational cost.…”

“…This is particularly difficult in larger more complex programs. To aid the diagnosis process, localisation techniques have been developed to highlight what code elements are particularly relevant to a functionality [2] or performance [3] defect. Finding where and how an issue manifests in source code can help indicate where a solution is likely to exist.…”

“…Profiling can be performed by instrumenting a program which increments a counter for each line of code each time it is executed. Additionally, a program can be executed with input of varying size to highlight what lines show an exponential increase in executing as input size increases [3]. Profiler techniques generally require developers must frequently trace back through a program to understand what code is contributing to a bottleneck [8].…”

“…As fault localisation [2] has been used to automate bug fixing [9], we seek similar methods for localising performance [3] to source code elements to benefit automated performance improvement [10], [11]. We seek localisation techniques which highlight code having the most effect on the overall execution cost of a program.…”

Mining hot calling contexts in small space

“…All the aforementioned works aim at associating performance metrics with distinct paths traversed either in the call graph or in the control flow graph during a program's execution. A different line of research explores input-sensitivity issues, with the goal of discovering workload-dependent performance bottlenecks (e.g., [46][47][48][49][50]). Context and input sensitivities represent two orthogonal aspects of program profiles, and the techniques described in this paper could be combined with input-sensitive profilers to pinpoint routines' calling contexts characterized by superlinear running times.…”