QTAccel: A Generic FPGA based Design for Q-Table based Reinforcement Learning Accelerators

Meng, Yuan; Kuppannagari, Sanmukh R.; Rajat, Rachit; Srivastava, Ajitesh; Kannan, Rajgopal; Prasanna, Viktor K.

doi:10.1109/ipdpsw50202.2020.00024

Cited by 13 publications

(23 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of integrating the environment into the learning system, states and rewards are seen as input to the hardware accelerator. The architecture contains one on-chip memory block per action in [48] the action space, which stores the Q-values of the respective action for each state. For a given state, the Q-values for all actions can be read at the same time.…”

Section: A Tabular Reinforcement Learningmentioning

confidence: 99%

“…Current publications focus on the implementation of Q-Learning but also suggest modifications to their proposed architectures to support SARSA [67,72,48]. The following paragraphs present similarities and differences between these architectures.…”

Section: A Classical Reinforcement Learningmentioning

confidence: 99%

“…In tabular reinforcement learning, an important design decision is how to represent the Q-table and how to compute the maximum Q-value for the future state. It can be saved continuously in one large on-chip memory block, as implemented in [48]. In this case, a Q max -table, storing the maximum Q-value for each state, is an efficient way to access these maximum values for the future state.…”

Section: Q-tablementioning

confidence: 99%

“…Two of the existing architectures use DSPs to implement the multiplications [67,48], while one architecture replaced the multiplications with barrel shifters to improve performance [72]. Another way to optimize the implementations is to introduce pipelining.…”

Section: Hardware Architecture Optimizationsmentioning

confidence: 99%

“…Another way to optimize the implementations is to introduce pipelining. Of the existing implementations, only [48] is a pipelined architecture, explaining its higher throughput compared to its predecessors and competitors.…”

Section: Hardware Architecture Optimizationsmentioning

confidence: 99%

See 4 more Smart Citations

A Survey of Domain-Specific Architectures for Reinforcement Learning

Rothmann

Porrmann

2022

IEEE Access

View full text Add to dashboard Cite

Reinforcement learning algorithms have been very successful at solving sequential decisionmaking problems in many different problem domains. However, their training is often time-consuming, with training times ranging from multiple hours to weeks. The development of domain-specific architectures for reinforcement learning promises faster computation times, decreased experiment turn-around time, and improved energy efficiency. This paper presents a review of hardware architectures for the acceleration of reinforcement learning algorithms. FPGA-based implementations are the focus of this work, but GPU-based approaches are considered as well. Both tabular and deep reinforcement learning algorithms are included in this survey. The techniques employed in different implementations are highlighted and compared. Finally, possible areas for future work are suggested, based on the preceding discussion of existing architectures.

show abstract

Section: A Tabular Reinforcement Learningmentioning

confidence: 99%

Section: A Classical Reinforcement Learningmentioning

confidence: 99%

Section: Q-tablementioning

confidence: 99%