Selecting benchmark combinations for the evaluation of multicore throughput

Velásquez, Ricardo A.; Michaud, Pierre; Seznec, André

doi:10.1109/ispass.2013.6557168

Cited by 11 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other work [20] examines the methods for selecting the single-program benchmark pairings-in our terminology, Bfrom which to define representative workloads. Selecting appropriate values of B is important, but it is orthogonal to our work.…”

Section: Impact Of S 0 Orderingmentioning

confidence: 99%

Multi-program benchmark definition

Jacobvitz

Hilton

Sorin

2015

2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)

View full text Add to dashboard Cite

Abstract-Although definition of single-program benchmarks is relatively straight-forward-a benchmark is a program plus a specific input-definition of multi-program benchmarks is more complex. Each program may have a different runtime and they may have different interactions depending on how they align with each other. While prior work has focused on sampling multiprogram benchmarks, little attention has been paid to defining the benchmarks in their entirety.In this work, we propose a four-tuple that formally defines multi-program benchmarks in a well-defined way. We then examine how four different classes of benchmarks created by varying the elements of this tuple align with real-world use-cases. We evaluate the impact of these variations on real hardware, and see drastic variations in results between different benchmarks constructed from the same programs. Notable differences include significant speedups versus slowdowns (e.g., +57% vs -5% or +26% vs -18%), and large differences in magnitude even when the results are in the same direction (e.g., 67% versus 11%).

show abstract

Section: Impact Of S 0 Orderingmentioning

confidence: 99%

Multi-program benchmark definition

Jacobvitz

Hilton

Sorin

2015

2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)

View full text Add to dashboard Cite

show abstract

“…Van Craeynest and Eeckhout [2011] show that for averaging throughput over the full population of possible coschedules, it may be more accurate to use a large random sample and fast approximate simulation than a small sample of manually chosen coschedules and detailed simulation. Velásquez et al [2013] show that fast approximate simulations may help select a representative sample of coschedules for detailed simulations.…”

Section: Related Workmentioning

confidence: 99%

“…Of course, this is practically not feasible, so only a sample of this space can be evaluated. Other studies have presented methods to select representative benchmark combinations and how to average throughput over multiple TPEXs [Velásquez et al 2013]. Our article specifically focuses on calculating the average throughput of one TPEX.…”

Section: Benchmark Suite Versus the Number Of Job Types In A Tpexmentioning

confidence: 99%

Multiprogram Throughput Metrics

Eyerman

Michaud

Rogiest

2014

ACM Trans. Archit. Code Optim.

Self Cite

View full text Add to dashboard Cite

Running multiple programs on a processor aims at increasing the throughput of that processor. However, defining meaningful throughput metrics in a simulation environment is not as straightforward as reporting execution time. This has led to an ongoing debate on what forms a meaningful throughput metric for multiprogram workloads. We present a method to construct throughput metrics in a systematic way: we start by expressing assumptions on job size, job distribution, scheduling, and so forth that together define a theoretical throughput experiment. The throughput metric is then the average throughput of this experiment. Different assumptions lead to different metrics, so one should be aware of these assumptions when making conclusions based on results using a specific metric.Throughput metrics should always be defined from explicit assumptions, because this leads to a better understanding of the implications and limits of the results obtained with that metric. We elaborate multiple metrics based on different assumptions. In particular, we identify the assumptions that lead to the commonly used weighted speedup and harmonic mean of speedups. Our study clarifies that they are actual throughput metrics, which was recently questioned. We also propose some new throughput metrics, which cannot always be expressed as a closed formula. We use real experimental data to characterize metrics and show how they relate to each other. ACM Reference Format:Stijn Eyerman, Pierre Michaud, and Wouter Rogiest. 2014. Multiprogram throughput metrics: A systematic approach. ACM Trans.

show abstract

“…For the heterogeneous multi-program workloads, we randomly construct 12 two-, 12 three-, 12 four-, etc., up to 12 twenty-four-thread combinations, while making sure that every benchmark is included an equal number of times for all thread counts. Velasquez et al [32] show that this balanced random sampling technique is more representative compared to fully random sampling.…”

Section: Workloadsmentioning

confidence: 99%

The benefit of SMT in the multi-core era

Eyerman

Eeckhout

2014

Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems

View full text Add to dashboard Cite

The number of active threads in a multi-core processor varies over time and is often much smaller than the number of supported hardware threads. This requires multi-core chip designs to balance core count and per-core performance. Low active thread counts benefit from a few big, highperformance cores, while high active thread counts benefit more from a sea of small, energy-efficient cores.This paper comprehensively studies the trade-offs in multi-core design given dynamically varying active thread counts. We find that, under these workload conditions, a homogeneous multi-core processor, consisting of a few highperformance SMT cores, typically outperforms heterogeneous multi-cores consisting of a mix of big and small cores (without SMT), within the same power budget. We also show that a homogeneous multi-core performs almost as well as a heterogeneous multi-core that also implements SMT, as well as a dynamic multi-core, while being less complex to design and verify. Further, heterogeneous multi-cores that power-gate idle cores yield (only) slightly better energyefficiency compared to homogeneous multi-cores.The overall conclusion is that the benefit of SMT in the multi-core era is to provide flexibility with respect to the available thread-level parallelism. Consequently, homogeneous multi-cores with big SMT cores are competitive highperformance, energy-efficient design points for workloads with dynamically varying active thread counts.

show abstract

Selecting benchmark combinations for the evaluation of multicore throughput

Cited by 11 publications

References 16 publications

Multi-program benchmark definition

Multi-program benchmark definition

Multiprogram Throughput Metrics

The benefit of SMT in the multi-core era

Contact Info

Product

Resources

About