Accuracy of performance counter measurements

Abstract

Many workload characterization studies depend on accurate measurements of the cost of executing a piece of code. Often these measurements are conducted using infrastructures to access hardware performance counters. Most modern processors provide such counters to count micro-architectural events such as retired instructions or clock cycles. These counters can be difficult to configure, may not be programmable or readable from user-level code, and can not discriminate between events caused by different software threads. Various software infrastructures address this problem, providing access to per-thread counters from application code. This paper constitutes the first comparative study of the accuracy of three commonly used measurement infrastructures (perfctr, perfmon2, and PAPI) on three common processors (Pentium D, Core 2 Duo, and AMD ATHLON 64 X2). We find significant differences in accuracy of various usage patterns for the different infrastructures and processors. Based on these results we provide guidelines for finding the best measurement approach.

Preliminary Investigation of Mobile System Features Potentially Relevant to HPC

“…In contrast, counts derived from simulations conducted using ValGrind [9] yielded consistent and reasonable values. Our experience is consistent with the observations of others who indicates performance counters can become inaccurate with more complex sequences of code such as those found within CoMD [15,16].…”

The ghost in the machine : don’t let it haunt your software performance measurements

“…To prevent the processor clock frequency from changing during mea surements, frequency scaling should be disabled [Zaparanuks et al, 2009]. Hopper [2013] points out that theoretical hardware performance values do not always reflect actual application performance due to many factors, including caching effects, data locality, and instruction sequences, among other things.…”

“…Zaparanuks et al [2009] conducted a well-designed experiment to determine the monitoring overhead of a number of configurations that allow user-level access to per-thread hardware counters in Linux. They observed that the monitoring overhead differs drastically between configurations.…”

“…Differing compiler optimization options can drastically change a program's performance [Flater and Guthrie, 2013;Zaparanuks et al, 2009]. Worse yet, code elimination can produce degenerate results.…”

Repeating History: Execution Replay for Parallel Haskell Programs

“…The most difficult problem these systems have to solve is that of shared memory interactions, something we can completely ignore because our source code is purely functional. In addition, some of these systems also try to replay the scheduling of threads (a requirement in our case), but they do so by using hardware counters [5,8], which makes them hardware dependent and subject to inaccurate measurements [12,17]. Another benefit of our approach is that we can modify the original program at will, as long as it produces the same allocations, in order to gather more information, and the replay will still be valid.…”