Ddper: Decentralized Distributed Prioritized Experience Replay

Liu, Sidun; Peng, Qiao; Dou, Yong; Li, Rongchun

doi:10.1109/icme51207.2021.9428188

Cited by 1 publication

(2 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an improvement method for sending more useful experiences, Liu et al [24] propose a distributed reinforcement learning architecture in which both actors and learner have a replay memory. The experience sampling is done twice in a hierarchical manner in both actors and learner.…”

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 99%

“…To address the heterogeneity of various computing platforms, it is expected that multiple buffer nodes are implemented on multiple edge servers; in this case, the number of experiences sampled from each buffer node should be tuned depending on the experience generation rate of each buffer node (e.g., more experiences should be sampled from a buffer node that produces more experiences). As the number of the buffer and actor nodes increases, the learner node would be a performance bottleneck; such a scalability issue can be addressed by introducing a hierarchical structure as in [24], and this direction is our future work.…”

Section: B Effectiveness and Scalability Of Proposed Approach For Rea...mentioning

confidence: 99%

See 1 more Smart Citation

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Morishima,

Matsutani

2024

IEEE Access

View full text Add to dashboard Cite

A computing cluster that interconnects multiple compute nodes is used to accelerate distributed reinforcement learning that uses DQN (Deep Q-Network). In distributed reinforcement learning, actor nodes acquire experiences by interacting with a given environment and a learner node optimizes the DQN model. When distributed reinforcement learning is used in practical applications such as robotics, we can assume that actor nodes are located in edge side while the learner node is located in cloud side. In this case, the long-haul communication between them imposes significant communication overheads. However, most prior works simply assume that actors and learner are located closely, and do not take the overheads into account. In this paper, we focus on the practical environment where the actors and learner are located remotely, and they interact via a buffer node that collects information from multiple actor nodes. We implement a prototype system in which the buffer and learner nodes are connected via a 25GbE (Gigabit Ethernet) switch and a 10km optical fiber cable. Although a replay memory functionality is closely associated with the learner side, in this paper we propose to combine the replay memory into the buffer node. In our experiments using the prototype system, the proposed approach is compared with an existing approach in terms of the training efficiency (i.e., training loss) and the transfer efficiency over the long-haul communication (i.e., average priority of transferred experiences). As a result, the training loss of the proposed approach is reduced to 26% of the existing approach, and the average priority is 3.92 times higher than the existing approach after the training loss is converged. These results demonstrate that the proposed approach can improve the training/communication efficiency compared with the existing approach in a practical system that imposes long-haul communication between the actors and learner.

show abstract

Section: Improvement Methods Of Transfer Efficiency Of Distributed Re...mentioning

confidence: 99%

Section: B Effectiveness and Scalability Of Proposed Approach For Rea...mentioning

confidence: 99%

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Morishima,

Matsutani

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

Ddper: Decentralized Distributed Prioritized Experience Replay

Cited by 1 publication

References 4 publications

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Contact Info

Product

Resources

About