EAIS: Energy-aware adaptive scheduling for CNN inference on high-performance GPUs

Yao, Chunrong; Liu, Wantao; Tang, Weiqing; Hu, Songlin

doi:10.1016/j.future.2022.01.004

Cited by 20 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This result is reasonable since the calculations in ML inference tasks are predefined. Meanwhile, profiling results also exhibit that the inference latency grows linearly with the batch size [9], [11], [13]. As shown in Fig.…”

Section: System Modelmentioning

confidence: 85%

“…Many researchers have conducted experiments to profile the computation latency of GPU when batch processing ML inference tasks [4], [7]- [11], [13], [18], [19]. It has been examined in [7] that the coefficient of variation (CV) of the batch processing time for image recognition is near CV=0, i.e.…”

Section: System Modelmentioning

confidence: 99%

“…(10) Since S is countable and the model can be easily verified to be unichain, according to [22], the optimality equations are (11) for ∀s ∈ S, where h(s) denotes the relative value of being in state s and g denotes the average cost per unit time. By Theorem 11.4.4 in [22], the constant and functions (g, h) satisfying (11) are exactly the optimal average cost per unit time and the corresponding relative value functions.…”

Section: [B]mentioning

confidence: 99%

“…Recently, several dynamic batching schemes have been proposed for machine learning inference systems [3], [4], [7]- [11]. Meanwhile, only a small amount of researches have made progresses in the performance analysis.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

Xu¹,

Sun²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

In up-to-date machine learning (ML) applications on cloud or edge computing platforms, batching is an important technique for providing efficient and economical services at scale. In particular, parallel computing resources on the platforms, such as graphics processing units (GPUs), have higher computational and energy efficiency with larger batch sizes. However, larger batch sizes may also result in longer response time, and thus it requires a judicious design. This paper aims to provide a dynamic batching policy that strikes a balance between efficiency and latency. The GPU-based inference service is modeled as a batch service queue with batch-size dependent processing time. Then, the design of dynamic batching is a continuous-time average-cost problem, and is formulated as a semi-Markov decision process (SMDP) with the objective of minimizing the weighted sum of average response time and average power consumption. The optimal policy is acquired by solving an associated discretetime Markov decision process (MDP) problem with finite state approximation and "discretization". By creatively introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Our results show that the optimal policies potentially possess a control limit structure. Numerical results also show that SMDP-based batching policies can adapt to different traffic intensities and outperform other benchmark policies. Furthermore, the proposed solution has notable flexibility in balancing power consumption and latency.

show abstract

Section: System Modelmentioning

confidence: 85%

Section: System Modelmentioning

confidence: 99%

Section: [B]mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

Xu¹,

Sun²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that research investigating power and energy consumption behaviors in modern clusters due to deep learning (DL) job processing is in its early stages. Yao et al [12] propose an energy-aware DL job scheduler to reduce the energy consumption of CNN inference services on GPU devices. Their scheduler coordinates batch processing and dynamic voltage frequency scaling (DVFS) to effectively respond to workload fluctuations, achieving a 28% reduction in energy consumption compared to its competitors while meeting latency service-level objectives.…”

Section: Introductionmentioning

confidence: 99%

Renewable-Aware Frequency Scaling Approach for Energy-Efficient Deep Learning Clusters

Park,

Kang

2024

Applied Sciences

View full text Add to dashboard Cite

Recently, renewable energy has emerged as an attractive means to reduce energy consumption costs for deep learning (DL) job processing in modern GPU-based clusters. In this paper, we propose a novel Renewable-Aware Frequency Scaling (RA-FS) approach for energy-efficient DL clusters. We have developed a real-time GPU core and memory frequency scaling method that finely tunes the training performance of DL jobs while maximizing renewable energy utilization. We introduce quantitative metrics: Deep Learning Job Requirement (DJR) and Deep Learning Job Completion per Slot (DJCS) to accurately evaluate the service quality of DL job processing. Additionally, we present a log-transformation technique to convert our non-convex optimization problem into a solvable one, ensuring the rigorous optimality of the derived solution. Through experiments involving deep neural network (DNN) model training jobs such as SqueezeNet, PreActResNet, and SEResNet on NVIDIA GPU devices like RTX3060, RTX3090, and RTX4090, we validate the superiority of our RA-FS approach. The experimental results show that our approach significantly improves performance requirement satisfaction by about 71% and renewable energy utilization by about 31% on average, compared to recent competitors.

show abstract

Scheduling Mechanisms in Serverless Computing

Ghobaei‐Arani

Ghorbian

2023

Lecture Notes on Data Engineering and Communications Technologies

View full text Add to dashboard Cite

EAIS: Energy-aware adaptive scheduling for CNN inference on high-performance GPUs

Cited by 20 publications

References 16 publications

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

Renewable-Aware Frequency Scaling Approach for Energy-Efficient Deep Learning Clusters

Scheduling Mechanisms in Serverless Computing

Contact Info

Product

Resources

About