Kelp: QoS for Accelerated Machine Learning Systems

Zhu, Haishan; Lo, David; Cheng, Liqun; Govindaraju, Rama; Ranganathan, Parthasarathy; Erez, Mattan

doi:10.1109/hpca.2019.00036

Cited by 18 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, they do not investigate the performance of existing serverless systems and pure serverless model serving systems while only working with AWS Lambda. Zhu et al [36] introduced Kelp, a software runtime that strives to isolate high-priority accelerated ML tasks from memory resource interference. They argue that in using accelerated machine learning, contention on host resources can significantly impact the efficiency of the accelerator.…”

Section: Related Workmentioning

confidence: 99%

MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms

Mahmoudi¹,

Khazaei²

2022

Preprint

View full text Add to dashboard Cite

Serving machine learning inference workloads on the cloud is still a challenging task on the production level. Optimal configuration of the inference workload to meet SLA requirements while optimizing the infrastructure costs is highly complicated due to the complex interaction between batch configuration, resource configurations, and variable arrival process. Serverless computing has emerged in recent years to automate most infrastructure management tasks.Workload batching has revealed the potential to improve the response time and cost-effectiveness of machine learning serving workloads. However, it has not yet been supported out of the box by serverless computing platforms. Our experiments have shown that for various machine learning workloads, batching can hugely improve the system's efficiency by reducing the processing overhead per request.In this work, we present MLProxy, an adaptive reverse proxy to support efficient machine learning serving workloads on serverless computing systems. MLProxy supports adaptive batching to ensure SLA compliance while optimizing serverless costs. We performed rigorous experiments on Knative to demonstrate the effectiveness of MLProxy. We showed that MLProxy could reduce the cost of serverless deployment by up to 92% while reducing SLA violations by up to 99% that can be generalized across state-of-the-art model serving frameworks.

show abstract

Section: Related Workmentioning

confidence: 99%

MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms

Mahmoudi¹,

Khazaei²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Zhang et al [8] propose a runtime system that exploits the newly added spatial multitasking feature in a GPU and raises the accelerator utilization while achieving the latency targets for user-facing services. Zhu et al [9] propose a software runtime that isolates high-priority accelerated machine learning tasks from memory resource interference. The OS-level scheduling methods commonly implement their design on a nonpreemptive accelerator design according to the modern GPU's execution model, which unavoidably makes them ineffective in some cases (priority inversion problem caused by the long-running kernels with lower priority, etc.).…”

Section: Related Workmentioning

confidence: 99%

Toward OS-Level and Device-Level Cooperative Scheduling for Multitasking GPUs

et al. 2020

View full text Add to dashboard Cite

As one of the most popular accelerators, the graphics processing unit (GPU) has been extensively adopted throughout the world. With the burst of new applications and the growing scale of data, co-running applications on limited GPU resources has become increasingly important due to its dramatic improvement in overall system efficiency. Quality of service (QoS) support among concurrent generalpurpose GPU (GPGPU) applications is currently one of the most trending research topics. Prior efforts have been focused on providing QoS support either with OS-level or device-level scheduling methods. Each of these scheduling methods possesses pros and cons and may be unable to independently cover all the scheduling cases. In this paper, we propose a cooperative QoS scheduling scheme (C-QoS) that consists of operating-system-level (OS-level) scheduling and device-level scheduling. Our proposed scheme can control the progress of a kernel and provide thorough QoS support for concurrent applications in multitasking GPUs. Due to the accurate resource management of the copy engine and execution engine, C-QoS achieves QoS goals 23.33% more often than state-of-the-art QoS support mechanisms. The results demonstrate that cooperative methods achieve 17.27% higher system utilization than uncooperative methods. INDEX TERMS Multitasking, parallel architectures, quality of service.

show abstract

“…Today's private datacenters host a diverse range of dataintensive applications, including machine learning training [7,22,38,51,60,62], SQL queries [9,30,58], graph processing [21,39,41], and big-data analytics [47,52]. Many of these applications are distributed and leverage parallel frameworks, such as Hadoop, Spark, Flink, and TensorFlow [1,8,17,53,61].…”

Section: Introductionmentioning

confidence: 99%

Saba: Rethinking Datacenter Network Allocation from Application's Perspective

Katebzadeh

Costa

Grot

2023

Proceedings of the Eighteenth European Conference on Computer Systems

View full text Add to dashboard Cite

Today's datacenter workloads increasingly comprise distributed data-intensive applications, including data analytics, graph processing, and machine-learning training. These applications are bandwidth-hungry and often congest the datacenter network, resulting in poor network performance, which hurts application completion time. Efforts made to address this problem generally aim to achieve max-min fairness at the flow or application level. We observe that splitting the bandwidth equally among workloads is sub-optimal for aggregate application-level performance because various workloads exhibit different sensitivity to network bandwidth: for some workloads, even a small reduction in the available bandwidth yields a significant increase in completion time; for others, the completion time is largely insensitive to the available bandwidth.Building on this insight, we propose Saba, an applicationaware bandwidth allocation framework that distributes network bandwidth based on application-level sensitivity. Saba combines ahead-of-time application profiling to determine bandwidth sensitivity with runtime bandwidth allocation using lightweight software support with no modifications to network hardware or protocols. Experiments with a 32server hardware testbed show that Saba improves average completion time by 1.88× (and by 1.27× in a simulated 1,944server cluster).

show abstract

Kelp: QoS for Accelerated Machine Learning Systems

Cited by 18 publications

References 38 publications

MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms

MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms

Toward OS-Level and Device-Level Cooperative Scheduling for Multitasking GPUs

Saba: Rethinking Datacenter Network Allocation from Application's Perspective

Contact Info

Product

Resources

About