Variance Reduction for Reinforcement Learning in Input-Driven Environments

Mao, Hongzi; Venkatakrishnan, Shaileshh Bojja; Schwarzkopf, Malte; Alizadeh, Mohammad

doi:10.48550/arxiv.1807.02264

Cited by 9 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The studies on applying reinforcement learning to non-Markovian environments is a longstanding research area [55], however, its applications in systems research and in particular resource management are not fully discovered. To the authors' knowledge the Mao et al [56] paper is the only research that has tackled one of the challenges of a semi-Markovian environment MDP in the area.…”

Section: Challenges Of Using Machine Learning For Resource Managementmentioning

confidence: 99%

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

Ghafouri¹,

Karami²,

Bakhtiarvand³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>In recent years Kubernetes has become the de facto standard in the realm of service orchestration. Despite its great benefits, there are still numerous challenges to make it compatible with decentralised cloud computing platforms. One of the challenges of mobile edge computing is that the location of the users is changing over time. This mobility will constantly alter the proximity of the users to their connected services. One solution to this problem is to regularly move services to computing nodes near the users. However, distributing the services in edge nodes only subject to user movements will result in the fragmentation of active nodes. This leads to having active nodes that do not use their full capacity. We have proposed a method called Mobile-Kube to reduce the latency of Kubernetes applications on mobile edge computing devices while maintaining energy consumption at a reasonable level. An experimental framework is designed on top of real-world Kubernetes clusters and real-world traces of mobile users’ movements have been used to simulate the user's mobility. Experimental results show that Mobile-Kube can achieve similar energy consumption performance to a heuristic approach that focuses on reducing energy consumption only while reducing the latency of services by 43%.</p>

show abstract

Section: Challenges Of Using Machine Learning For Resource Managementmentioning

confidence: 99%

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

Ghafouri¹,

Karami²,

Bakhtiarvand³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Section: Challenges Of Using Machine Learning For Resource Managementmentioning

confidence: 99%

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

Ghafouri

Karami

Bakhtiarvan

et al. 2022

2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing (UCC)

View full text Add to dashboard Cite

In recent years Kubernetes has become the de facto standard in the realm of service orchestration. Despite its great benefits, there are still numerous challenges to make it compatible with decentralised cloud computing platforms. One of the challenges of mobile edge computing is that the location of the users is changing over time. This mobility will constantly alter the proximity of the users to their connected services. One solution to this problem is to regularly move services to computing nodes near the users. However, distributing the services in edge nodes only subject to user movements will result in the fragmentation of active nodes. This leads to having active nodes that do not use their full capacity. We have proposed a method called Mobile-Kube to reduce the latency of Kubernetes applications on mobile edge computing devices while maintaining energy consumption at a reasonable level. An experimental framework is designed on top of real-world Kubernetes clusters and real-world traces of mobile users' movements have been used to simulate the users' mobility. Experimental results show that Mobile-Kube can achieve similar energy consumption performance to a heuristic approach that focuses on reducing energy consumption only while reducing the latency of services by 43%.

show abstract

“…Meanwhile, subtracting a baseline remains a heuristic [26] that has strong empirical but limited theoretical support. One possible benefit of a baseline is that it provides variance reduction [10], which has motivated work on designing alternative baselines that further reduce variance [30,4,20,31]. However, other work [7] has shown that variance reduction is not necessarily aligned with policy learning quality.…”

Section: Introductionmentioning

confidence: 99%

The Role of Baselines in Policy Gradient Optimization

Mei¹,

Chung²,

Thomas³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study the effect of baselines in on-policy stochastic policy gradient optimization, and close the gap between the theory and practice of policy optimization methods. Our first contribution is to show that the state value baseline allows on-policy stochastic natural policy gradient (NPG) to converge to a globally optimal policy at an O(1/t) rate, which was not previously known. The analysis relies on two novel findings: the expected progress of the NPG update satisfies a stochastic version of the non-uniform Łojasiewicz (NŁ) inequality, and with probability 1 the state value baseline prevents the optimal action's probability from vanishing, thus ensuring sufficient exploration. Importantly, these results provide a new understanding of the role of baselines in stochastic policy gradient: by showing that the variance of natural policy gradient estimates remains unbounded with or without a baseline, we find that variance reduction cannot explain their utility in this setting. Instead, the analysis reveals that the primary effect of the value baseline is to reduce the aggressiveness of the updates rather than their variance. That is, we demonstrate that a finite variance is not necessary for almost sure convergence of stochastic NPG, while controlling update aggressiveness is both necessary and sufficient. Additional experimental results verify these theoretical findings.

show abstract

Variance Reduction for Reinforcement Learning in Input-Driven Environments

Cited by 9 publications

References 19 publications

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

Mobile-Kube: Mobility-aware and Energy-efficient Service Orchestration on Kubernetes Edge Servers

The Role of Baselines in Policy Gradient Optimization

Contact Info

Product

Resources

About