Discriminating Equivalent Algorithms via Relative Performance

Abstract

For a given linear algebra problem, we consider those solution algorithms that are mathematically equivalent to one another, and that mostly consist of a sequence of calls to kernels from optimized libraries such as BLAS and LAPACK.Although equivalent (at least in exact precision), those algorithms typically exhibit significant differences in terms of performance, and naturally, we are interested in finding the fastest one(s). In practice, we often observe that multiple algorithms yield comparable performance characteristics. Therefore, we aim to identify the subset of algorithms that are reliably faster than the rest. To this end, instead of quantifying the performance of an algorithm in absolute terms, we present a measurement-based approach that assigns a relative score to the algorithms in comparison to one another. The relative performance is encoded by sorting the algorithms based on pair-wise comparisons and ranking them into equivalence classes, where more than one algorithm can obtain the same rank. We show that the relative performance leads to robust identification of the fastest algorithms, that is, reliable identifications even with noisy system conditions.

Performance Comparison for Scientific Computations on the Edge via Relative Performance

“…Instead of summarizing the performance statistic (such as mean or minimum execution time) of all the N measurements into one number, multiple statistics are evaluated and compared on data that is randomly sampled from the N measurements; this approach is commonly known as "bootstrapping". This allows us to gain more information from the set of measurements (also referred as distributions or histogram) of execution times, which are then used for performance comparisons [15]. The distributions are compared pair-wise and merged into the same cluster if the comparison of two algorithms evaluates to be performance-equivalent.…”

“…Although we considered a use-case where it is possible to execute and measure all the different combination of solution algorithms, this may not be an ideal solution for applications where there is an exponential number of alternative implementations. For instance, the linear algebra expression in the line 4 of Procedure 6 can alone have many different equivalent algorithms, each having a different sequence of calls to optimized libraries such as BLAS and LAPACK [44]; typically these algorithms also show significant difference in performance, even without considering the split of computation among devices [15] [45]. Therefore, in case of exponential explosion of the search space, our methodology can still be applied on a subset of possible solutions and the resulting clusters with relative scores can be used as a ground truth to guide the search of algorithm via reinforcement learning.…”

“…The intermediate steps of the bubble sort procedure for this illustration is shown in Figure 2. In every iteration of the sort procedure, adjacent pairs of algorithms alg s[j] and alg s[j+1] , starting from the right-most algorithm in S, are compared using the bootstrap strategy from [15] Computing the relative scores: The clustering procedure discussed above is not deterministic, especially when the fluctuations in the performance measurements are large and the distributions are partially overlapping (for example, distributions "AD" and "AA" in Figure 1b). Such overlaps become more evident when the number of measurements N is small.…”

“…Contributions: We use the methodology we developed in [15] to cluster a given set of equivalent algorithms A. In contrast to our analysis in [15], where we aimed to identify only the subset of "fastest" algorithms, we now focus on clusters of algorithms from all performance classes.…”

“…Contributions: We use the methodology we developed in [15] to cluster a given set of equivalent algorithms A. In contrast to our analysis in [15], where we aimed to identify only the subset of "fastest" algorithms, we now focus on clusters of algorithms from all performance classes. One of the reasons for looking beyond just the fastest algorithms is that, in an edge computing environment, one might be interested in switching between algorithms from different performance classes to optimize energy in a device; for instance, when energy consumption of a particular device reaches a certain threshold, one might be interested in switching to an algorithm that performs fewer floating point operations (FLOPs) on that device, and then switches back to the high-performance algorithm after a while.…”