Active Learning in Performance Analysis

Duplyakin, Dmitry; Brown, Jed; Ricci, Robert

doi:10.1109/cluster.2016.63

Cited by 25 publications

(15 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More information on this method is available in [27]. The use of GP for modeling software performance has been proposed in [14]. We also use GP here for verification on our datasets.…”

Section: Modeling Methodsmentioning

confidence: 99%

“…Sumo [16] Generic ++ The authors in [14] recommend the use of Gaussian Processes (GP) as a flexible method to both model performance and choose the next configurations to sample. Our tests show however that the trends witnessed in VNF profiling are not efficiently captured by GPs.…”

Section: Sampling Heuristic Drawbacksmentioning

confidence: 99%

“…It is worth noting that uniform sampling, exemplified above, is not the least performing method we encountered during our tests. When following the approach outlined in [14], a Gaussian Process (GP) is used to select online which samples to measure next. In fact, GP is a modeling method, but it also produces an estimate of uncertainty in the prediction, a confidence interval for the predicted values as indicated in Fig.…”

Section: Unsuccessful Strategiesmentioning

confidence: 99%

See 2 more Smart Citations

Optimized Sampling Strategies to Model the Performance of Virtualized Network Functions

et al. 2020

View full text Add to dashboard Cite

Modern network services make increasing use of virtualized compute and network resources. This is enabled by the growing availability of softwarized network functions, which take on major roles in the total traffic flow (such as caching, routing or as firewall). To ensure reliable operation of its services, the service provider needs a good understanding of the performance of the deployed softwarized network functions. Ideally, the service performance should be predictable, given a certain input workload and a set of allocated (virtualized) resources (such as vCPUs and bandwidth). This helps to estimate more accurately how much resources are needed to operate the service within its performance specifications. To predict its performance, the network function should be profiled in the whole range of possible input workloads and resource configurations. However, this input can span a large space of multiple parameters and many combinations to test, resulting in an expensive and overextended measurement period. To mitigate this, we present a profiling framework and a sampling heuristic to help select both workload and resource configurations to test. Additionally, we compare several machine-learning based methods for the best prediction accuracy, in combination with the sampling heuristic. This work has been performed in the framework of the NG-PaaS and 5GTANGO project, funded by the European Commission under the Horizon 2020 and 5G-PPP Phase2 programmes, resp. under Grant Agreement No. 761 557 and 761 493 (http://ngpaas.eu) (https://www.5gtango.eu). This work is partly funded by UGent BOF/GOA project "Autonomic Networked Multimedia Systems".

show abstract

“…More information on this method is available in [27]. The use of GP for modeling software performance has been proposed in [14]. We also use GP here for verification on our datasets.…”

Section: Modeling Methodsmentioning

confidence: 99%

Section: Sampling Heuristic Drawbacksmentioning

confidence: 99%

Section: Unsuccessful Strategiesmentioning

confidence: 99%

See 1 more Smart Citation

Optimized Sampling Strategies to Model the Performance of Virtualized Network Functions

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Online data analysis and reduction co-design is complicated by a much larger configuration space than is found in many conventional application design problems. In the case of a single application, the number of configurable parameters is often small, and thus, we can identify good values for configuration parameters via a mix of performance modeling and experiments (Foster 1994; Hoefler et al, 2011; Duplyakin et al, 2016; Balaprakash et al, 2018). In contrast, consider an ODAR code that couples one or more simulation, analysis, and reduction applications.…”

Section: Odar Co-design Processmentioning

confidence: 99%

Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Foster

Ainsworth

Bessac

et al. 2021

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

show abstract

“…Other approaches combine traditional modeling techniques with methods from machine learning, including active and transfer learning, neural networks or decision trees, to further improve the robustness of their predictions [32]. Duplyakin et al, for example, apply active learning to suggest followup experiments that can help refine their initial performance models created by Gaussian process regression [33]. We, in contrast, employ reinforcement learning to derive a parameter value selection heuristic that is efficient for all kinds of HPC applications, not tuned for one specific application.…”

Section: Related Workmentioning

confidence: 99%

Learning Cost-Effective Sampling Strategies for Empirical Performance Modeling

Ritter

Calotoiu

Rinke

et al. 2020

2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Identifying scalability bottlenecks in parallel applications is a vital but also laborious and expensive task. Empirical performance models have proven to be helpful to find such limitations, though they require a set of experiments in order to gain valuable insights. Therefore, the experiment design determines the quality and cost of the models. Extra-P is an empirical modeling tool that uses small-scale experiments to assess the scalability of applications. Its current version requires an exponential number of experiments per model parameter. This makes the creation of empirical performance models very expensive, and in some situations even impractical. In this paper, we propose a novel parameter-value selection heuristic, which functions as a guideline for the experiment design, leveraging sparse performance-modeling, a technique that only needs a polynomial number of experiments per model parameter. Using synthetic analysis and data from three different case studies, we show that our solution reduces the average modeling costs by about 85% while retaining 92% of the model accuracy.

show abstract

Active Learning in Performance Analysis

Cited by 25 publications

References 5 publications

Optimized Sampling Strategies to Model the Performance of Virtualized Network Functions

Optimized Sampling Strategies to Model the Performance of Virtualized Network Functions

Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Learning Cost-Effective Sampling Strategies for Empirical Performance Modeling

Contact Info

Product

Resources

About