Learning Cooperative Oversubscription for Cloud by Chance-Constrained Multi-Agent Reinforcement Learning

Sheng, Junjie; Rajmohan, Saravan; Yang, Fangkai; Qiao, Bo; Dong, Hang; Wang, Xiangfeng; Jin, Bo; Wang, Jun; Qin, Si; Rajmohan, Saravan; Lin, Qingwei; Zhang, Dongmei

doi:10.1145/3543507.3583298

Cited by 2 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…What is the post-convergence stability of CO I N in presence of uncertainty as opposed to vanilla IL? Baselines: In this paper, we compare our approach against the following baselines (1) Grid search with different oversubscription probabilities, where all the subscribers have the same oversubscription rate (2) Vanilla IL such as Behavior Cloning or BC, (3) Policy gradient reinforcement learning such as DDPG (Lillicrap et al 2015) (4) Multi-Agent reinforcement learning or MA (Sheng et al 2022) and ( 5) IL with hard constraints. Most existing resource management or oversubscription frameworks in practice (Salahuddin, Al-Fuqaha, and Guizani 2016;Gosavi, Bandla, and Das 2002;Kumbhare et al 2021;Shihab et al 2019;Lawhead and Gosavi 2019) exploit traditional reinforcement learning as the optimization strategy.…”

Section: Methodsmentioning

confidence: 99%

NENYA: Cascade Reinforcement Learning for Cost-Aware Failure Mitigation at Microsoft 365

Wang

Zhao

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

We address the challenge of learning safe and robust decision policies in presence of uncertainty in context of the real scientific problem of adaptive resource oversubscription to enhance resource efficiency while ensuring safety against resource congestion risk. Traditional supervised prediction or forecasting models are ineffective in learning adaptive policies whereas standard online optimization or reinforcement learning is difficult to deploy on real systems. Offline methods such as imitation learning (IL) are ideal since we can directly leverage historical resource usage telemetry. But, the underlying aleatoric uncertainty in such telemetry is a critical bottleneck. We solve this with our proposed novel chanceconstrained imitation learning framework, which ensures implicit safety against uncertainty in a principled manner via a combination of stochastic (chance) constraints on resource congestion risk and ensemble value functions. This leads to substantial (≈ 3 − 4×) improvement in resource efficiency and safety in many oversubscription scenarios, including resource management in cloud services.

show abstract