Character-level recurrent neural networks in practice: comparing training and sampling schemes

Abstract

Recurrent neural networks are nowadays successfully used in an abundance of applications, going from text, speech and image processing to recommender systems. Backpropagation through time is the algorithm that is commonly used to train these networks on specific tasks. Many deep learning frameworks have their own implementation of training and sampling procedures for recurrent neural networks, while there are in fact multiple other possibilities to choose from and other parameters to tune. In existing literature this is very often overlooked or ignored. In this paper we therefore give an overview of possible training and sampling schemes for character-level recurrent neural networks to solve the task of predicting the next token in a given sequence. We test these different schemes on a variety of datasets, neural network architectures and parameter settings, and formulate a number of take-home recommendations. The choice of training and sampling scheme turns out to be subject to a number of trade-offs, such as training stability, sampling time, model performance and implementation effort, but is largely independent of the data. Perhaps the most surprising result is that transferring hidden states for correctly initializing the model on subsequences often leads to unstable training behavior depending on the dataset.

EmoTwiCS: a corpus for modelling emotion trajectories in Dutch customer service dialogues on Twitter

“…Another recurrencebased approach to ERC is called COSMIC (Ghosal et al, 2020), a knowledge-based model that is related to DialogueRNN in its network structure and that adds information about, e.g., causal relations, mental states, and actions to improve performance. Even though these recurrence-based models can in theory handle infinitely long sequences, in practice long-term contextual information is not always propagated due to vanishing gradients and practical limitations in recurrent depth when applying backpropagation-through-time (Pascanu et al, 2013;De Boom et al, 2019).…”

Natural Language Enhancement for English Teaching Using Character-Level Recurrent Neural Network with Back Propagation Neural Network based Classification by Deep Learning Architectures

“…analysis, text analysis, speech recognition, and other fields, providing solutions to a variety of real-world problems [Sun et al 2021]. The basic knowledge of transfer learning, numerous types of methodologies utilised to accomplish transfer learning, as well as how transfer learning was applied in many subfields of medical image analysis were all evaluated by the authors [Boom et al 2019]. The review demonstrates that recent developments in DL, particularly advances in transfer learning, have enabled the identification, classification, and quantification of specific patterns from a large number of medical pictures [Hwang and Sung, 2017].…”

Image Hashtag Recommendations Using a Voting Deep Neural Network and Associative Rules Mining Approach

“…An ensemble of deep convolutional neural networks was also used by Reference [ 9 ] in a similar task; however, it was a single-label classification problem. A single-label approach was also presented in Reference [ 10 ], where the authors explored the possibility of generating hashtags for an input image and leveraged it to generate meaningful anecdotes connected to the essence of the image by applying a character-level Recurrent (RNN) Neural Network [ 11 , 12 ].…”