Evaluating goodness-of-fit of discrete distribution models in quantitative linguistics

Mačutek, Ján; Wimmer, Gejza

doi:10.1080/09296174.2013.799912

Cited by 35 publications

(15 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The obtained values of R 2 appear to be quite close to unity meaning the fit is good (usually, R 2 > 0.9 is considered acceptable [22]). For instance, the Chinese and Lojban translations of Alice in Wonderland yield R 2 > 0.98 and R 2 > 0.99, respectively.…”

Section: Resultsmentioning

confidence: 75%

See 1 more Smart Citation

Models of Frequency Spectrum in Texts Based on Quantum Distributions in Fractional Space Dimensions

Rovenchak

2015

2015 20th International Conference on Control Systems and Computer Science

View full text Add to dashboard Cite

From the analogy between the frequency structure of texts and the Bose-distribution in statistical physics the parameters are calculated to describe texts in several languages. Using the model of harmonic oscillators, the frequency spectrum of text can be fitted to the occupation numbers in the Bose-distribution via the analog of temperature and the space dimension. Best results are achieved for highly analytic languages (Chinese, Hawaiian, Vietnamese, Mauritian Creole, Lojban) with fractional space dimensions D < 1.

show abstract

Section: Resultsmentioning

confidence: 75%

“…The goodness-of-fit can be tested using the determination coefficient R 2 defined by [22]: In this expression, N exp is the mean value of observed N exp j . In our case, the summations run over all the js starting from 2 as j = 1 is fixed by z.…”

Section: Resultsmentioning

confidence: 99%

Models of Frequency Spectrum in Texts Based on Quantum Distributions in Fractional Space Dimensions

Rovenchak

2015

2015 20th International Conference on Control Systems and Computer Science

View full text Add to dashboard Cite

show abstract

“…As a rule of thumb, ≤ 0.02 indicates a good fit; a "more tolerant" version of the rule accepts a good fit of a model if ≤ 0.05 (cf. Mačutek and Wimmer, 2013, where also other possibilities how to avoid the problem of large samples are mentioned). Parameters were estimated by the minimum 2 method (cf.…”

Section: Language Materials and Methodologymentioning

confidence: 99%

Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019)

2019

View full text Add to dashboard Cite

The use of corpora for NLP and linguistics has only increased in recent years. In NLP, machine learning systems are by nature data-intensive, and in linguistics there is a renewed interest in the empirical validation of linguistic theory, particularly through corpus evidence. While the first statistical parsers have long been trained on the Penn treebank phrase structures, dependency treebanks, whether natively annotated with dependencies, or converted from phrase structures, have become more and more popular, as evidenced by the success of the Universal Dependency project, currently uniting 120 treebanks in 80 languages, annotated in the same dependency-based scheme. The availability of these resources has boosted empirical quantitative studies in syntax. It has also lead to a growing interest in theoretical questions around syntactic dependency, its history, its foundations, and the analyses of various constructions in dependency-based frameworks. Furthermore, the availability of large, multilingual annotated data sets, such as those provided by the Universal Dependencies project, has made cross-linguistic analysis possible to an extent that could only be dreamt of only a few years ago. In this context it was natural to bring together TLT (Treebanks and Linguistic Theories), the historical conference on treebanks as linguistic resources, Depling (The international conference on Dependency Linguistics), the conference uniting research on models and theories around dependency representations, and UDW (Universal Dependency Workshop), the annual meeting of the UD project itself. Moreover, in order to create a point of contact with the large community working in quantitative linguistics it seemed expedient to create a workshop dedicated to quantitative syntactic measures on treebanks and raw corpora, which gave rise to Quasy, the first workshop on Quantitative Syntax. And this led us to the first SyntaxFest. Because the potential audience and submissions to the four events were likely to have substantial overlap, we decided to have a single reviewing process for the whole SyntaxFest. Authors could choose to submit their paper to one or several of the four events, and in case of acceptance, the program co-chairs would decide which event to assign the accepted paper to.

show abstract

“…problem is the chi-square goodness-of-fit test, whose reliability is increasingly doubted by linguists (Wang, 2012;Mačutek & Wimmer, 2013). It is inadequate if the sample size is very large, but no criterion with respect to what "large" means is given so far.…”

Section: Rank-frequency Distribution Of Parts Of Speechmentioning

confidence: 99%

Part-of-speech studies in Chinese

Wang

2016

Journal of Quantitative Linguistics

View full text Add to dashboard Cite

This paper studies parts of speech in Chinese on data taken from the Modern Chinese Dictionary (5th edition). First, the part-of-speech polyfunctionality (ambiguity) of words is determined; then the corresponding distribution and rank-frequency sequence are analysed. The Waring and right truncated modified Zipf-Alekseev distributions are successfully fitted to the data. Second, the 121 patterns that the total 3742 polyfunctional words yield are presented. The polyfunctionality of patterns distributes according to the positive Cohen-binomial distribution, while the rank-frequency sequence abides by the negative hypergeometric distribution. Third, we discuss the mechanism behind the polyfunctionality phenomenon: Chinese words diversify in the dimension of their function but not in their form as can be expected from an analytic language. The Popescu-Altmann function captures the distribution of the variants of each part of speech. Fourth, we analyse the polyfunctionality distributions of individual parts of speech. Out of the 12 parts of speech which the dictionary distinguishes, six can be modelled by the Poisson distribution, four by the mixed Poisson, and two by the Singh-Poisson distribution. In order to obtain a general form, we apply the mixed Poisson distribution for all the parts of speech by controlling one parameter. We make a first attempt to plot the polyfunctionality distributions of individual parts of speech in Ord's system, which surprisingly shows approximately a hyperbola.

show abstract

Evaluating goodness-of-fit of discrete distribution models in quantitative linguistics

Cited by 35 publications

References 14 publications

Models of Frequency Spectrum in Texts Based on Quantum Distributions in Fractional Space Dimensions

Models of Frequency Spectrum in Texts Based on Quantum Distributions in Fractional Space Dimensions

Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019)

Part-of-speech studies in Chinese

Contact Info

Product

Resources

About