A Rigorous Benchmarking and Performance Analysis Methodology for Python Workloads

Abstract

Computer architecture and computer systems research and development is heavily driven by benchmarking and performance analysis. It is thus of paramount importance that rigorous methodologies are used to draw correct conclusions and steer research and development in the right direction. While rigorous methodologies are widely used for native and managed programming language workloads, scripting language workloads are subject to ad-hoc methodologies which lead to incorrect and misleading conclusions. In particular, we find incorrect public statements regarding different virtual machines for Python, the most popular scripting language. The incorrect conclusion is a result of using the geometric mean speedup and not making a distinction between start-up and steady-state performance. In this paper, we propose a statistically rigorous benchmarking and performance analysis methodology for Python workloads, which makes a distinction between start-up and steady-state performance and which summarizes average performance across a set of benchmarks using the harmonic mean speedup. We find that a rigorous methodology makes a difference in practice. In particular, we find that the PyPy JIT compiler outperforms the CPython interpreter by 1.76× for steady-state while being 2% slower for start-up, which refutes the statement on the PyPy website that 'PyPy outperforms CPython by 4.4× on average' based on the geometric mean speedup and not making a distinction between start-up and steady-state. We use the proposed methodology to analyze Python workloads which yields several interesting findings regarding PyPy versus CPython performance, start-up versus steady-state performance, the impact of a workload's input size, and Python workload execution characteristics at the microarchitecture level.

Faster or Slower? Performance Mystery of Python Idioms Unveiled with Empirical Evidence

“…The measurement of execution time of non-idiomatic and idiomatic code is far from trivial due to the non-determinism such as Python Virtual Machine (VM) and garbage collector [7]. To overcome such non-determinism, researchers repeatedly execute the code multiple times [7], [5], [8], [26], [27], [28], [10], [9]. Since different VM invocations may result in different code execution time, and multiple executions of the code in a VM invocation may vary, we execute each code 35 iterations on 50 VM invocations as done in previous studies [7], [8], [9], [10].…”

“…To overcome such non-determinism, researchers repeatedly execute the code multiple times [7], [5], [8], [26], [27], [28], [10], [9]. Since different VM invocations may result in different code execution time, and multiple executions of the code in a VM invocation may vary, we execute each code 35 iterations on 50 VM invocations as done in previous studies [7], [8], [9], [10]. Since first iterations (i.e., warm-up iterations) are subject to noise caused by library loading, measurements are only collected in iterations that are subsequent to warmup.…”

“…To measure and compare the code execution time reliably, we execute a piece of code in 50 Virtual Machine (VM) invocations and collect the execution time of 35 execution (excluding the first 3 warm-up executions) in each VM invocation, following the process in previous studies [7], [8], [9], [10], [11]. We perform the bootstrapping with hierarchical random re-sampling and replacement on both VM invocations and execution iterations, following previous studies [12], [13], [8], [14].…”

Python meets JIT compilers: A simple implementation and a comparative evaluation

“…However, researchers have not yet reached a consensus on averaging the speedups of multiple benchmarks. Some 31 endorse the geometric mean, some 24,32 advocate the harmonic mean, and some 33 argue that the choice does not matter. Thus, we show both the geometric and harmonic mean speedup over time in Figure 10.…”

“…The full suite or subsets thereof are also broadly recognized by relevant researchers. [23][24][25][26][27][28] These benchmarks exhibit varying characteristics, as they focus on different application scenarios: arithmetic calibration, HTML template rendering, regular expression processing, object serialization, database accessing and so forth. Some of these benchmarks are Python-dominant, primarily leveraging pure Python code to complete their computational tasks, while some are native-dominant, drawing heavily on functionalities implemented in native code provided by the interpreter or extensions.…”