Model-Free reinforcement learning with continuous action in practice

Degris, Thomas; Pilarski, Patrick M.; Sutton, Richard S.

doi:10.1109/acc.2012.6315022

Cited by 198 publications

(132 citation statements)

References 15 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…Many machine learning approaches base their optimization on variations of policy gradients (see for instance Degris et al (2012); Watkins and Dayan (1992)). We use the simplest policy gradient formulation, corresponding to the seminal work by Sutton et al (1999).…”

Section: Policy Gradient Methods On Loss Function For Max-cut Problemmentioning

confidence: 99%

Experimental performance of graph neural networks on random instances of max-cut

Yao¹,

Bandeira²,

Villar³

2019

Wavelets and Sparsity XVIII

View full text Add to dashboard Cite

This note explores the applicability of unsupervised machine learning techniques towards hard optimization problems on random inputs. In particular we consider Graph Neural Networks (GNNs) -a class of neural networks designed to learn functions on graphs -and we apply them to the max-cut problem on random regular graphs. We focus on the max-cut problem on random regular graphs because it is a fundamental problem that has been widely studied. In particular, even though there is no known explicit solution to compare the output of our algorithm to, we can leverage the known asymptotics of the optimal max-cut value in order to evaluate the performance of the GNNs.In order to put the performance of the GNNs in context, we compare it with the classical semidefinite relaxation approach by Goemans and Williamson (SDP), and with extremal optimization, which is a local optimization heuristic from the statistical physics literature. The numerical results we obtain indicate that, surprisingly, Graph Neural Networks attain comparable performance to the Goemans and Williamson SDP. We also observe that extremal optimization consistently outperforms the other two methods. Furthermore, the performances of the three methods present similar patterns, that is, for sparser, and for larger graphs, the size of the found cuts are closer to the asymptotic optimal max-cut value.

show abstract

Section: Policy Gradient Methods On Loss Function For Max-cut Problemmentioning

confidence: 99%

Experimental performance of graph neural networks on random instances of max-cut

Yao¹,

Bandeira²,

Villar³

2019

Wavelets and Sparsity XVIII

View full text Add to dashboard Cite

show abstract

“…In an off-policy setting, actor-critic estimates the value function of π θ (a|s) by averaging the state distribution of behavior policy β(a|s) [11]. Instead of considering the stochastic policy π θ (s |a), the deterministic policy gradient (DPG) theorem [32] proves that policy gradient framework can be extended to find deterministic off-policy µ θ (s), which is given as follows:…”

Section: Supervisor Of Clinician Decisionmentioning

confidence: 99%

Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

Wang

Zhang

et al. 2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

186

112

View full text Add to dashboard Cite

Dynamic treatment recommendation systems based on large-scale electronic health records (EHRs) become a key to successfully improve practical clinical outcomes. Prior relevant studies recommend treatments either use supervised learning (e.g. matching the indicator signal which denotes doctor prescriptions), or reinforcement learning (e.g. maximizing evaluation signal which indicates cumulative reward from survival rates). However, none of these studies have considered to combine the benefits of supervised learning and reinforcement learning. In this paper, we propose Supervised Reinforcement Learning with Recurrent Neural Network (SRL-RNN), which fuses them into a synergistic learning framework. Specifically, SRL-RNN applies an off-policy actor-critic framework to handle complex relations among multiple medications, diseases and individual characteristics. The "actor" in the framework is adjusted by both the indicator signal and evaluation signal to ensure effective prescription and low mortality. RNN is further utilized to solve the Partially-Observed Markov Decision Process (POMDP) problem due to the lack of fully observed states in real world applications. Experiments on the publicly real-world dataset, i.e., MIMIC-3, illustrate that our model can reduce the estimated mortality, while providing promising accuracy in matching doctors' prescriptions.

show abstract

“…We begin by examining the choice of employing deep reinforcement learning against simpler reinforcement learning algorithms for solving the problem formulated in §IV-B. We compare the performance of Iris when using DDPG against the stochastic policy gradient algorithm of [82], which employs linear function approximators for the actor and the critic (Lin-PG). As illustrated in Fig.…”

Section: B Deep Learning Benefits Feasibility and Scalabilitymentioning

confidence: 99%

Iris: Deep Reinforcement Learning Driven Shared Spectrum Access Architecture for Indoor Neutral-Host Small Cells

Foukas

Marina

Kontovasilis

2019

IEEE J. Select. Areas Commun.

View full text Add to dashboard Cite

We consider indoor mobile access, a vital use case for current and future mobile networks. For this key use case, we outline a vision that combines a neutral-host based shared small-cell infrastructure with a common pool of spectrum for dynamic sharing as a way forward to proliferate indoor small-cell deployments and open up the mobile operator ecosystem. Towards this vision, we focus on the challenges pertaining to managing access to shared spectrum (e.g., 3.5GHz US CBRS spectrum). We propose Iris, a practical shared spectrum access architecture for indoor neutral-host small-cells. At the core of Iris is a deep reinforcement learning based dynamic pricing mechanism that efficiently mediates access to shared spectrum for diverse operators in a way that provides incentives for operators and the neutral-host alike.We then present the Iris system architecture that embeds this dynamic pricing mechanism alongside cloud-RAN and RAN slicing design principles in a practical neutral-host design tailored for the indoor small-cell environment. Using a prototype implementation of the Iris system, we present extensive experimental evaluation results that not only offer insight into the Iris dynamic pricing process and its superiority over alternative approaches but also demonstrate its deployment feasibility. Index TermsIndoor mobile access, small cells, neutral host, RAN slicing, C-RAN, shared spectrum, dynamic pricing, deep reinforcement learning.

show abstract

Model-Free reinforcement learning with continuous action in practice

Cited by 198 publications

References 15 publications

Experimental performance of graph neural networks on random instances of max-cut

Experimental performance of graph neural networks on random instances of max-cut

Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

Iris: Deep Reinforcement Learning Driven Shared Spectrum Access Architecture for Indoor Neutral-Host Small Cells

Contact Info

Product

Resources

About