Accelerating Distributed Deep Reinforcement Learning by In-Network Experience Sampling

Furukawa, Masaki; Matsutani, Hiroki

doi:10.1109/pdp55904.2022.00020

Cited by 2 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overall, our proposed approach is classified as an improvement method for sending more useful experiences to improve the transfer efficiency in the edge-cloud settings, but the prior works mentioned above do not focus on edge-cloud implementation where actors and learner's communication is long-haul as assumed in this paper. Although our previous work [12] also focused on such an edge-cloud environment as mentioned in Section 1, this paper is quite different from the previous work since the previous work proposed an acceleration method of communication by DPDK.…”

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 92%

“…In this case, gradient aggregation between computers becomes a performance bottleneck; thus, they propose to accelerate this process by performing the gradient aggregation within network switches. Our past work [12] also proposes an acceleration method of communication between actors and learner in Ape-X style architecture using DPDK and F-Stack.…”

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 99%

“…The first approach is accelerating the communica-tion by network optimization techniques. Our previous work adopts this approach using DPDK and F-Stack [12]. The second approach is sending useful experiences preferentially for the training.…”

Section: Introductionmentioning

confidence: 99%

“…This paper is an extended version of our conference paper[12] by adding implementations of the proposed and existing architectures in a practical assumption and evaluations of the proposed approach in terms of training and transfer efficiency compared with the existing approach.2 VOLUME 4, 2016 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication.…”

mentioning

confidence: 99%

See 3 more Smart Citations

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Morishima,

Matsutani

2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

A computing cluster that interconnects multiple compute nodes is used to accelerate distributed reinforcement learning that uses DQN (Deep Q-Network). In distributed reinforcement learning, actor nodes acquire experiences by interacting with a given environment and a learner node optimizes the DQN model. When distributed reinforcement learning is used in practical applications such as robotics, we can assume that actor nodes are located in edge side while the learner node is located in cloud side. In this case, the long-haul communication between them imposes significant communication overheads. However, most prior works simply assume that actors and learner are located closely, and do not take the overheads into account. In this paper, we focus on the practical environment where the actors and learner are located remotely, and they interact via a buffer node that collects information from multiple actor nodes. We implement a prototype system in which the buffer and learner nodes are connected via a 25GbE (Gigabit Ethernet) switch and a 10km optical fiber cable. Although a replay memory functionality is closely associated with the learner side, in this paper we propose to combine the replay memory into the buffer node. In our experiments using the prototype system, the proposed approach is compared with an existing approach in terms of the training efficiency (i.e., training loss) and the transfer efficiency over the long-haul communication (i.e., average priority of transferred experiences). As a result, the training loss of the proposed approach is reduced to 26% of the existing approach, and the average priority is 3.92 times higher than the existing approach after the training loss is converged. These results demonstrate that the proposed approach can improve the training/communication efficiency compared with the existing approach in a practical system that imposes long-haul communication between the actors and learner.

show abstract

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 92%

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Morishima,

Matsutani

2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…The DQN is a reinforcement learning algorithm that combines deep neural networks with the Q-learning algorithm. The utilization of a profound neural network is employed for the purpose of forecasting the Q-value function, a theoretical construct that has been investigated [6]. The learning process of this network involves the minimization of the discrepancy between the anticipated and intended Q-values.…”

Section: Dqnmentioning

confidence: 99%

Comparison of Deep Q-Learning Network and Double Deep Q-Learning Network for Trading Strategy

Chen

2024

Proceedings of the 4th International Conference on Economic Management and Big Data Applications, ICEMBDA 2023, October 27–29,

View full text Add to dashboard Cite

This study introduces a unique program designed to provide a specialized trading environment for the purpose of training agents. Two principal agents, namely Deep Q-Leaning Network (DQN) and Double DQN (DDQN), were taught and afterwards evaluated, with their performance measured using profit metrics, providing insights into their effectiveness in formulating stock policies. The comparative examination of these agents, based on their respective training periods, consistently demonstrated the higher performance of DQN in comparison to DDQN. As for the progress into the future, there are several potential optimizations that might be considered. These optimizations encompass fine-tuning the elements of neural networks, extending the duration of training sessions, and integrating emerging brain designs and training paradigms. The suggested application demonstrates versatility and adaptability by offering opportunities to incorporate a wide range of financial market data sets. This feature holds the potential to provide a more comprehensive training environment and im-prove the trading techniques employed by the agents.

show abstract

Accelerating Distributed Deep Reinforcement Learning by In-Network Experience Sampling

Cited by 2 publications

References 2 publications

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Comparison of Deep Q-Learning Network and Double Deep Q-Learning Network for Trading Strategy

Contact Info

Product

Resources

About