Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Abstract

Neural networks can achieve impressive performance on many natural language processing applications, but they typically need large labeled data for training and are not easily interpretable. On the other hand, symbolic rules such as regular expressions are interpretable, require no training, and often achieve decent accuracy; but rules cannot benefit from labeled data when available and hence underperform neural networks in rich-resource scenarios. In this paper, we propose a type of recurrent neural networks called FA-RNNs that combine the advantages of neural networks and regular expression rules. An FA-RNN can be converted from regular expressions and deployed in zero-shot and cold-start scenarios. It can also utilize labeled data for training to achieve improved prediction accuracy. After training, an FA-RNN often remains interpretable and can be converted back into regular expressions. We apply FA-RNNs to text classification and observe that FA-RNNs significantly outperform previous neural approaches in both zeroshot and low-resource settings and remain very competitive in rich-resource settings.

Neuralizing Regular Expressions for Slot Filling

“…An i-FST is much more compact and faster but it still: (1) has too many parameters (especially T ∈ R V ×K×K ) compared to a traditional BiRNN; and (2) is unable to incorporate external word embeddings. To tackle these problems, we adopt the tensor decomposition-based method proposed by Jiang et al (2020) and modify the forward and backward score computation accordingly (Steps 3 and 4 in Algorithm 2).…”

“…For static word embeddings such as GloVe, we again adopt the method of Jiang et al (2020). Let E w ∈ R V ×D denote the word embedding matrix we want to incorporate.…”

“…Our work is inspired by Jiang et al (2020) who neuralize REs for text classification. Apart from the difference in the output forms, our method differs from theirs mainly in that: (1) we target the slot filling task and hence neuralize REs with capturing groups; (2) we convert REs into an FST that produces output sequences, while they convert REs into a finite state automaton (FSA) that decides acceptance of strings; (3) the tensor parameter of an FST has higher order than an FSA, which leads to a more complicated procedure to reduce the computational complexity; (4) our decoding algorithm has to invoke both forward and backward processes, while theirs requires only a forward process.…”

Exploring Distantly-Labeled Rationales in Neural Network Models

“…Luo et al (2018) concatenates information of regular expressions to word embeddings for spoken language understanding. Jiang et al (2020) uses an RNN to model regular expressions for text classification tasks. Most of these works provide effective ways to utilize word-level knowledge, but none of them formally considers the quality issues with the distantly-labeled rationales.…”

“…Recent studies have shown an increasing interest in incorporating human knowledge into neural network models (Xu et al, 2018;Vashishth et al, 2018;Luo et al, 2018;Li and Srikumar, 2019;Jiang et al, 2020). For many natural language processing (NLP) tasks, such domain knowledge often refers to salient words annotated by human experts, which are also called rationales.…”