Large scale recurrent neural network on GPU

Li, Boxun; Zhou, Erjin; Huang, Bo; Duan, Jiayi; Wang, Yu; Xu, Ning; Zhang, Jiaxing; Yang, Huazhong

doi:10.1109/ijcnn.2014.6889433

Cited by 55 publications

(21 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previously, Li et al proposed a pipeline architecture to improve the parallelism of RNN [15]. As illustrated in Figure 4, it partitions the feed-forward phase into two stages: the data flow from input to hidden layer represented by gray boxes and the computation from hidden to output layer denoted in white boxes.…”

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

“…Thus, the computation complexity of h(t) is mainly determined Figure 4. The data flow of a two-stage pipelined RNN structure [15]. The computation complexity of white boxes in output layer is much bigger than that of gray boxes in hidden layer.…”

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

“…Thus, the execution time of the second stage is much longer than that of the first one. Such a pipelined structure [15] is not optimal for the entire workload.…”

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

“…While Output Layer I is in operation at t − 2, the hidden layer will submit more data to the next available output layer processing element, e.g., Output Layer II. As such, the speed-up ratio of the proposed design over the two-stage pipelined structure [15] can be approximated by…”

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

“…The implementation led to extremely high costs in system development and power consumption: a typical voice search language model for US English stream trained on 230B words consumes 1KJ energy per search [13]. The hardware acceleration, thus, is necessary and implementations in ASICs [14], GPUs [15] and FPGAs [16] have been explored. Among them, FPGA based accelerators have attracted great attentions for flexible reconfiguration capability and high energy efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

FPGA Acceleration of Recurrent Neural Network Based Language Model

et al. 2015

2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines

Self Cite

View full text Add to dashboard Cite

Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2% in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.

show abstract

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

“…Thus, the execution time of the second stage is much longer than that of the first one. Such a pipelined structure [15] is not optimal for the entire workload.…”

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

Section: A Increase Parallelism Between Hidden and Output Layersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

FPGA Acceleration of Recurrent Neural Network Based Language Model

et al. 2015

2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines

Self Cite

View full text Add to dashboard Cite

show abstract

Convolutional ProteinUnetLM competitive with long short‐term memory‐based protein secondary structure predictors

et al. 2022

View full text Add to dashboard Cite

The protein secondary structure (SS) prediction plays an important role in the characterization of general protein structure and function. In recent years, a new generation of algorithms for SS prediction based on embeddings from protein language models (pLMs) is emerging. These algorithms reach state-of-the-art accuracy without the need for time-consuming multiple sequence alignment (MSA) calculations. Long short-term memory (LSTM)-based SPOT-1D-LM and NetSurfP-3.0 are the latest examples of such predictors. We present the ProteinUnetLM model using a convolutional Attention U-Net architecture that provides prediction quality and inference times at least as good as the best LSTM-based models for 8-class SS prediction (SS8).Additionally, we address the issue of the heavily imbalanced nature of the SS8 problem by extending the loss function with the Matthews correlation coefficient, and by proper assessment using previously introduced adjusted geometric mean (AGM) metric. ProteinUnetLM achieved better AGM and sequence overlap score than LSTMbased predictors, especially for the rare structures 310-helix (G), beta-bridge (B), and high curvature loop (S). It is also competitive on challenging datasets without homologs, free-modeling targets, and chameleon sequences. Moreover, ProteinUnetLM outperformed its previous MSA-based version ProteinUnet2, and provided better AGM than AlphaFold2 for 1/3 of proteins from the CASP14 dataset, proving its potential for making a significant step forward in the domain. To facilitate the usage of our solution by protein scientists, we provide an easy-to-use web interface under https://biolib.com/SUT/ProteinUnetLM/.

show abstract

A survey on graphic processing unit computing for large‐scale data mining

Cano

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

General purpose computation using Graphic Processing Units (GPUs) is a wellestablished research area focusing on high-performance computing solutions for massively parallelizable and time-consuming problems. Classical methodologies in machine learning and data mining cannot handle processing of massive and high-speed volumes of information in the context of the big data era. GPUs have successfully improved the scalability of data mining algorithms to address significantly larger dataset sizes in many application areas. The popularization of distributed computing frameworks for big data mining opens up new opportunities for transformative solutions combining GPUs and distributed frameworks. This survey analyzes current trends in the use of GPU computing for large-scale data mining, discusses GPU architecture advantages for handling volume and velocity of data, identifies limitation factors hampering the scalability of the problems, and discusses open issues and future directions.

show abstract

Large scale recurrent neural network on GPU

Cited by 55 publications

References 13 publications

FPGA Acceleration of Recurrent Neural Network Based Language Model

FPGA Acceleration of Recurrent Neural Network Based Language Model

Convolutional ProteinUnetLM competitive with long short‐term memory‐based protein secondary structure predictors

A survey on graphic processing unit computing for large‐scale data mining

Contact Info

Product

Resources

About