A Case Study of Performance Degradation Attributable to Run-Time Bounds Checks on C++ Vector Access

Abstract

Programmers routinely omit run-time safety checks from applications because they assume that these safety checks would degrade performance. The simplest example is the use of arrays or array-like data structures that do not enforce the constraint that indices must be within bounds. This report documents an attempt to measure the performance penalty incurred by two different implementations of bounds-checking in C and C++ using a simple benchmark and a desktop PC with a modern superscalar CPU. The benchmark consisted of a loop that wrote to array elements in sequential order. With this configuration, relative to the best performance observed for any access method in C or C++, mean degradation of only (0.881 ± 0.009) % was measured for a standard bounds-checking access method in C++. This case study showed the need for further work to develop and refine measurement methods and to perform more comparisons of this type. Comparisons across different use cases, configurations, programming languages, and environments are needed to determine under what circumstances (if any) the performance advantage of unchecked access is actually sufficient to outweigh the negative consequences for security and software quality.

Estimation of Uncertainty in Application Profiles

“…An extended example that applies both the Welch-Sattherthwaite formula and theŠidák inequality to software performance measurements can be found in Ref. [25].…”

“…• Controlled variables -Dell Precision T5400 PC as used in [25], fixed CPU frequency Each run of the test program produced a value for both levels of N by dividing execution time between two functions, with the main program being overhead. The order of tests progressed upward through each level of self-time fragmentation before starting on the next of the 1000 iterations.…”

The Case for Writing Network Drivers in High-Level Programming Languages

“…However, the Rust code only requires 6% (11%) more cycles per packet overall despite doing more work. Synthetic benchmarks can achieve an even lower overhead of bounds checking [18]. A modern superscalar out of order processor can effectively hide the overhead introduced by these safety checks: normal execution does not trigger bounds check violations, the processor is therefore able to correctly predict (branch mispredict rate is at 0.2% -0.3%) and speculatively execute the correct path.…”

The ghost in the machine : don’t let it haunt your software performance measurements

“…Mytkowicz et al [2009] observed that measurement bias is commonplace in published papers with experimental results while Flater and Guthrie [2013] noted that the earlier observation by Berry [1992] that statistical design of experiments for the analysis of computer system performance is not applied as often as it ought still seems true more than a decade later. This observation was supported by Vitek and Kalibera [2012] who stated that papers in proceedings of computer science conferences regularly appear without a comparison to the state of the art, without appropriate benchmarks, without any mention of limitations, and without sufficient detail to reproduce the experiments.…”

“…Differing compiler optimization options can drastically change a program's performance [Flater and Guthrie, 2013;Zaparanuks et al, 2009]. Worse yet, code elimination can produce degenerate results.…”