Extending Zipf’s law to n-grams for large corpora

Ha, Le Quan; Hanna, Philip; Ji, Ming; Smith, Fraser

doi:10.1007/s10462-009-9135-4

Cited by 24 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is not the case for a specific written language corpus, but for about any written language corpus. Indeed, Zipf's law has been found across many languages and language groups [19,34,36] and has not only been found for word frequencies, but also for other linguistic phenomena, such as part-of-speech tags [36], -grams [18] and number words [11]. What's more, Zipf's law has even been found for non-human communication, such as for dolphin whistles [32] and gesturing in gorillas [16].…”

Section: Introductionmentioning

confidence: 99%

Zipf's Law in Human-Machine Dialog

Linders

Louwerse

2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

Zipf's law is a mathematically relatively simple formula stating that the frequency of a word is inversely correlated with its rank. Zipf's law is well-known in computational linguistics and cognitive sciences alike. In the context of agent development, however, Zipf's law has hardly ever been mentioned. This is surprising as principles regarding language likely benefit the development of conversational agents. This paper serves as a starting point to explore the role of Zipf's law in agent development, showing that Zipf's law also applies to dialog. Moreover, it can shed light on human-machine dialog. In addition to word frequency distributions that demonstrate Zipf's law, we also included frequency distributions of words at specific positions in the sentence as well as turn lengths. Zipf's law was found in the far majority of analyses we conducted. In addition, we investigated whether Zipf's law can be used to detect differences between human and agent-generated speech through correlating the distributions and found that even though both the human and agent frequency distributions follow Zipf's law, these distributions are not necessarily similar, shedding light on where agent dialog may distinguish itself from human dialog. The findings in this paper can thus serve as a way to monitor to what extent ubiquitous patterns in human-human dialog are found in humanmachine dialog. CCS CONCEPTS • Computing methodologies → Discourse, dialogue and pragmatics; • Mathematics of computing → Distribution functions.

show abstract

Section: Introductionmentioning

confidence: 99%

Zipf's Law in Human-Machine Dialog

Linders

Louwerse

2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

show abstract

“…This phenomenon is also present when investigating lemmas [6]. Similarly, n-gram phrases fit this distribution [7,8]. Therefore, to some extent, repetitions of identical n-grams are likely to be found in large corpora.…”

Section: Reuse Of Partial Resultsmentioning

confidence: 89%

Speeding up Natural Language Parsing by Reusing Partial Results

Strzyz¹,

Gómez-Rodríguez²

2019

EasyChair Preprints

View full text Add to dashboard Cite

This paper proposes a novel technique that applies casebased reasoning in order to generate templates for reusable parse tree fragments, based on PoS tags of bigrams and trigrams that demonstrate low variability in their syntactic analyses from prior data. The aim of this approach is to improve the speed of dependency parsers by avoiding redundant calculations. This can be resolved by applying the predefined templates that capture results of previous syntactic analyses and directly assigning the stored structure to a new n-gram that matches one of the templates, instead of parsing a similar text fragment again. The study shows that using a heuristic approach to select and reuse the partial results increases parsing speed by reducing the input length to be processed by a parser. The increase in parsing speed comes at some expense of accuracy. Experiments on English show promising results: the input dimension can be reduced by more than 20% at the cost of less than 3 points of Unlabeled Attachment Score.

show abstract

“…Baayen (2008, p. 226) elaborates the problem of sample independence of Zipf's law. In fact, Ha, Hanna, Ming, and Smith (2009) propose an extension of Zipf's law for large corpora. Egghe (2000) shows that the rank-frequency distribution follows Zipf's law with an additional exponent.…”

Section: Examining the N-gram Distributionsmentioning

confidence: 99%

ThetextcatPackage forn-Gram Based Text Categorization inR

Feinerer¹,

Buchta²,

Geiger³

et al. 2013

J. Stat. Soft.

View full text Add to dashboard Cite

Identifying the language used will typically be the first step in most natural language processing tasks. Among the wide variety of language identification methods discussed in the literature, the ones employing the Cavnar and Trenkle (1994) approach to text categorization based on character n-gram frequencies have been particularly successful. This paper presents the R extension package textcat for n-gram based text categorization which implements both the Cavnar and Trenkle approach as well as a reduced n-gram approach designed to remove redundancies of the original approach. A multi-lingual corpus obtained from the Wikipedia pages available on a selection of topics is used to illustrate the functionality of the package and the performance of the provided language identification methods.

show abstract

Extending Zipf’s law to n-grams for large corpora

Cited by 24 publications

References 24 publications

Zipf's Law in Human-Machine Dialog

Zipf's Law in Human-Machine Dialog

Speeding up Natural Language Parsing by Reusing Partial Results

ThetextcatPackage forn-Gram Based Text Categorization inR

Contact Info

Product

Resources

About