Frequency Consolidation Among Word N-Grams

Buerki, Andreas

doi:10.1007/978-3-319-69805-2_30

Cited by 3 publications

(3 citation statements)

References 22 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark. N-grams should not be calculated over sentence boundaries (Buerki 2017), but Huston, Moffat, and Croft (2011) showed that using sentence boundaries as separation can greatly reduce the number of 4-grams produced. Since the majority of longer n-grams will be dropped by the thresholding, empirically it does not matter whether sentence boundaries are considered in the n-gramming process.…”

Section: Multiword Expressions: Practical Significancementioning

confidence: 99%

Comparison of text preprocessing methods

Chai

2022

Nat. Lang. Eng.

View full text Add to dashboard Cite

Text preprocessing is not only an essential step to prepare the corpus for modeling but also a key area that directly affects the natural language processing (NLP) application results. For instance, precise tokenization increases the accuracy of part-of-speech (POS) tagging, and retaining multiword expressions improves reasoning and machine translation. The text corpus needs to be appropriately preprocessed before it is ready to serve as the input to computer models. The preprocessing requirements depend on both the nature of the corpus and the NLP application itself, that is, what researchers would like to achieve from analyzing the data. Conventional text preprocessing practices generally suffice, but there exist situations where the text preprocessing needs to be customized for better analysis results. Hence, we discuss the pros and cons of several common text preprocessing methods: removing formatting, tokenization, text normalization, handling punctuation, removing stopwords, stemming and lemmatization, n-gramming, and identifying multiword expressions. Then, we provide examples of text datasets which require special preprocessing and how previous researchers handled the challenge. We expect this article to be a starting guideline on how to select and fine-tune text preprocessing methods.

show abstract

Section: Multiword Expressions: Practical Significancementioning

confidence: 99%

Comparison of text preprocessing methods

Chai

2022

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

“…very high-frequency) words. 5 In step 2, the various lengths of identified sequences had their frequencies consolidated and were combined into a single list using Sub-String (Buerki 2017). At step 3, lexico-structural filters were applied to the lists of sequences to remove sequences that were likely to lack semantic unity.…”

Section: Identification Of Flmentioning

confidence: 99%

(How) is Formulaic Language Universal? Insights from Korean, German and English

Buerki¹

2020

Formulaic Language and New Data

Self Cite

View full text Add to dashboard Cite

Items of formulaic language, also referred to as phraseological units or common turns of phrase, are in evidence in a very large number of languages. However, the extent to which languages feature such formulaic material is unclear. Similarly, how formulaicity may be understood across typologically different languages and whether indeed there is a concept of formulaic language that applies across languages, are questions which have not generally been discussed. Using a novel data set consisting of topically matched corpora in three typologically different languages (Korean, German and English), this study proposes an empirically founded universal concept for formulaic language and discusses what the shape of this concept implies for the theoretical understanding of formulaic language going forward. In particular, it is argued that the nexus of the concept of formulaic language cannot be fixed at any particular structural level (such as the phrase or the level of polylexicality) and incorporates elements specified at varying levels of schematicity. This means that a cross-linguistic concept of formulaic language fits in well with a constructionist view of linguistic structure. Andreas Buerki disgrace), multi-word terms (open letter, contempt of court) as well as other habitual sequences (half an hour, no chance of X, behind closed doors) and, to the extent to which they are in recurrent use within a community, idioms (like get one's knickers in a twist) and even proverbs (garbage in, garbage out). FL is held to be of central importance to the functioning of language in a number of key ways. For example, besides making up a sizable portion of language in use (Altenberg 1998; Butler 2005: 223), knowledge of FL is thought a prerequisite for full proficiency in a language, register, dialect or sociolect. This is because habitual turns of phrase are crucially only a subset of all expressions that might be judged grammatical (e.g. Bally 1909: 73; Pawley and Syder 1983: 191; O'Keeffe et al. 2007: 60) and so knowledge of the boundaries of grammaticality alone is insufficient. FL is also thought to ease processing load during language production and thus it is nothing less than a key enabler of fluency in lan

show abstract

“…In the final step of the procedure, the frequencies of n-grams of various lengths are consolidated such that shorter sequences that are included in longer sequences are not counted multiple times, and sequences that only occur as part of longer sequences are eliminated (cf. Buerki, 2017).…”

Section: Datamentioning

confidence: 99%

Furiously fast: On the speed of change in formulaic language

Buerki

2019

Yearbook of Phraseology

Self Cite

View full text Add to dashboard Cite

Addressing a topic that has been marginal to discussions within historical linguistics, this study looks at how extent and speed of language change can be quantified meaningfully using corpus data. Looking specifically at formulaic language (understood here as word sequences that instantiate typical phrasings), a solidly data-based assessment of the speed of change within a 100-year time window is offered. This includes both a relative determination of speed (against the speed of change in lexis which is generally thought to be the fastest type of linguistic change, cf. Algeo 1980: 264; Trask and Millar 2010: 7) as well as a new independent measure of speed which is easy to interpret and therefore of high validity, while also robust and potentially applicable to any linguistic feature that can be counted in corpus data. Using data from a diachronic reference corpus of 20th century German, it is shown that change in formulaic language is very notably faster than lexical change, that the extent of change over a century is comparable in extent to contemporary inter-genre variation and that overall, the rate of change does fluctuate somewhat at the level of temporal granularity employed in this study. It is also argued that quantifying the speed of linguistic change can play an important role in building a deeper understanding of language change in general.

show abstract

Frequency Consolidation Among Word N-Grams

Cited by 3 publications

References 22 publications

Comparison of text preprocessing methods

Comparison of text preprocessing methods

(How) is Formulaic Language Universal? Insights from Korean, German and English

Furiously fast: On the speed of change in formulaic language

Contact Info

Product

Resources

About