Morphology Matters: A Multilingual Language Modeling Analysis

Park, Hyunji Hayley; Zhang, Katherine J.; Haley, Coleman; Steimel, Kenneth; Liu, Han; Schwartz, Lane

doi:10.1162/tacl_a_00365

Cited by 11 publications

(8 citation statements)

References 17 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that the tokenization choice could act as an inductive bias for seq2seq models, and character-level framing could be useful even for tasks that are not truly character-level. This observation also aligns with the findings of the recent work on language modeling complexity (Park et al, 2021;Mielke et al, 2019). For many languages, including several Slavic ones related to the Serbian-Bosnian pair, a character-level language model yields lower surprisal than the one trained on BPE units, suggesting that the effect might also be explained by the character tokenization making the language easier to language-model.…”

Section: Results and Analysissupporting

confidence: 89%

Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction

Ryskina¹,

Hovy²,

Berg-Kirkpatrick³

et al. 2021

Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

View full text Add to dashboard Cite

Traditionally, character-level transduction problems have been solved with finite-state models designed to encode structural and linguistic knowledge of the underlying process, whereas recent approaches rely on the power and flexibility of sequence-to-sequence models with attention. Focusing on the less explored unsupervised learning scenario, we compare the two model classes side by side and find that they tend to make different types of errors even when achieving comparable performance. We analyze the distributions of different error classes using two unsupervised tasks as testbeds: converting informally romanized text into the native script of its language (for Russian, Arabic, and Kannada) and translating between a pair of closely related languages (Serbian and Bosnian). Finally, we investigate how combining finite-state and sequence-to-sequence models at decoding time affects the output quantitatively and qualitatively. 1

show abstract

Section: Results and Analysissupporting

confidence: 89%

Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction

Ryskina¹,

Hovy²,

Berg-Kirkpatrick³

et al. 2021

Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

View full text Add to dashboard Cite

show abstract

“…The sentencepiece library was used in our research article. Next, the text is segmented into variable-length subword units using unigram language modeling ( Park et al, 2021 ). The subword units are then sorted based on their frequency of occurrence in the corpus , and a predefined number of units are selected to form the final vocabulary.…”

Section: Methodsmentioning

confidence: 99%

Deep learning-based automatic action extraction from structured chemical synthesis procedures

Vaškevičius,

Kapočiūtė-Dzikienė,

Vaškevičius

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

This article proposes a methodology that uses machine learning algorithms to extract actions from structured chemical synthesis procedures, thereby bridging the gap between chemistry and natural language processing. The proposed pipeline combines ML algorithms and scripts to extract relevant data from USPTO and EPO patents, which helps transform experimental procedures into structured actions. This pipeline includes two primary tasks: classifying patent paragraphs to select chemical procedures and converting chemical procedure sentences into a structured, simplified format. We employ artificial neural networks such as long short-term memory, bidirectional LSTMs, transformers, and fine-tuned T5. Our results show that the bidirectional LSTM classifier achieved the highest accuracy of 0.939 in the first task, while the Transformer model attained the highest BLEU score of 0.951 in the second task. The developed pipeline enables the creation of a dataset of chemical reactions and their procedures in a structured format, facilitating the application of AI-based approaches to streamline synthetic pathways, predict reaction outcomes, and optimize experimental conditions. Furthermore, the developed pipeline allows for creating a structured dataset of chemical reactions and procedures, making it easier for researchers to access and utilize the valuable information in synthesis procedures.

show abstract

“…The LM establishes the statistical relationship between words in the language. Word-level statistical modeling of morphologically complex languages can not achieve the word sequence prediction capabilities of simple morphology languages [8,9]. Additionally, the finite-sized word vocabulary of a pronunciation lexicon does not cover complex word forms and loan words that appear in a real-world setting.…”

Section: Deep Neural Network-hidden Markov Model (Dnn-hmm)-based Asr ...mentioning

confidence: 99%

“…A syllabification algorithm tailored for Malayalam script using finite state transducers has been proposed in [34]. The linguistic rules for syllable tokenization described in Table 6 have been computationally implemented as in Algorithm 1 and made available in the Mlphon Python library 9 . The algorithm analyzes the input text sequence and determines whether it falls into one of the four allowable categories of syllable structures in Malayalam.…”

Section: Syllable Tokensmentioning

confidence: 99%

See 1 more Smart Citation

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Manohar,

A R,

Rajan

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond $$n=3$$ n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

show abstract

Morphology Matters: A Multilingual Language Modeling Analysis

Cited by 11 publications

References 17 publications

Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction

Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction

Deep learning-based automatic action extraction from structured chemical synthesis procedures

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Contact Info

Product

Resources

About