Character and Subword-Based Word Representation for Neural Language
            Modeling Prediction

Labeau, Matthieu; Allauzen, Alexandre

doi:10.18653/v1/w17-4101

Cited by 10 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The structure of these subword sequences is especially important towards the representation of nonverbal dynamics. In addition, modeling subword information has become essential for various tasks in natural language processing (Faruqui et al 2017), including language modeling (Labeau and Allauzen 2017;), learning word representations for different languages (Peters et al 2018;Oh et al 2018;Bojanowski et al 2016), and machine translation (Kudo 2018;Sennrich, Haddow, and Birch 2015). However, many of these previous works in understanding and modeling multimodal language has ignored the role of subword analysis.…”

Section: Introductionmentioning

confidence: 99%

Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Wang

Shen

Liu

et al. 2019

AAAI

318

176

View full text Add to dashboard Cite

Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication. Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions. As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear. To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. In addition, we seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. To this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN) that models the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues. Our proposed model achieves competitive performance on two publicly available datasets for multimodal sentiment analysis and emotion recognition. We also visualize the shifted word representations in different nonverbal contexts and summarize common patterns regarding multimodal variations of word representations. Negative-shifted word representation Original word representation Positive-shifted word representation Visual Acoustic ⋯ excited voice raised eyebrows Visual Acoustic ⋯ soft voice shock Word Representation Space

show abstract

Section: Introductionmentioning

confidence: 99%

Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Wang

Shen

Liu

et al. 2019

AAAI

318

176

View full text Add to dashboard Cite

show abstract

“…Secondly, we would like to introduce characterlevel information in our models, based on (Wieting et al, 2016;Labeau and Allauzen, 2017), in order to overcome the problem of out-of-vocabulary (OOV) words and learn syntactic and stylistic features (Peters et al, 2018), which are highly indicative of emotions and their intensity.…”

Section: Discussionmentioning

confidence: 99%

NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning

Baziotis

Nikolaos²,

Chronopoulou

et al. 2018

Proceedings of the 12th International Workshop on Semantic Evaluation

View full text Add to dashboard Cite

In this paper we present deep-learning models that submitted to the SemEval-2018 Task 1 competition: "Affect in Tweets". We participated in all subtasks for English tweets. We propose a Bi-LSTM architecture equipped with a multi-layer self attention mechanism. The attention mechanism improves the model performance and allows us to identify salient words in tweets, as well as gain insight into the models making them more interpretable. Our model utilizes a set of word2vec word embeddings trained on a large collection of 550 million Twitter messages, augmented by a set of word affective features. Due to the limited amount of task-specific training data, we opted for a transfer learning approach by pretraining the Bi-LSTMs on the dataset of Semeval 2017, Task 4A. The proposed approach ranked 1 st in Subtask E "Multi-Label Emotion Classification", 2 nd in Subtask A "Emotion Intensity Regression" and achieved competitive results in other subtasks.

show abstract

“…However, this naive and standard way ignores the language information of the text. In order to effectively resort to linguistic information for scene text recognition, we incorporate subword [75] tokenization mechanism in NLP [76] into the text recognition method. Subword tokenization algorithms aim to decompose rare words into meaningful subwords and remain frequently used words, so that the grammatical information of word has already been captured in the subwords.…”

Section: Multi-granularity Predictionsmentioning

confidence: 99%

Multi-granularity Prediction for Scene Text Recognition

Wang

Cheng

Yao

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Due to the enormous technical challenges and wide range of applications, scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this tough problem, numerous innovative methods have been successively proposed, and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet functionally powerful vision STR model, which is built upon ViT and a tailored Adaptive Addressing and Aggregation (A 3 ) module. It already outperforms most previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. To produce the final recognition results, two strategies for effectively fusing the multi-granularity predictions are devised: the first strategy, called Confidence-based Fusion Strategy (CFS), employs a straightforward rule based on the confidence scores of the multi-granularity predictions, while the second one, called Learnable Fusion Strategy (LFS), is realized with a trainable module to directly measure the cross-modal similarities between text (predicted words) and images, akin to CLIP. The resultant algorithm (termed MGP-STR) is able to push the performance envelope of STR to an even higher level. Specifically, MGP-STR with LFS achieves an average recognition accuracy of 94% on standard benchmarks for scene text recognition (such as IIIT 5K-word, ICDAR 2015, SVT and CUTE). Moreover, it also achieves state-of-the-art results on widely-used handwritten benchmarks (IAM, CVL and RIMES) as well as more challenging scene text datasets (ArT, COCO-Text and Uber-Text), demonstrating the generality of the proposed MGP-STR algorithm. The source code and models will be available at: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR.

show abstract

Character and Subword-Based Word Representation for Neural Language Modeling Prediction

Cited by 10 publications

References 36 publications

Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning

Multi-granularity Prediction for Scene Text Recognition

Contact Info

Product

Resources

About