The word entropy of natural languages

Bentz, Christian; Alikaniotis, Dimitrios

doi:10.48550/arxiv.1606.06996

Cited by 4 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Entropy We also consider measuring the amount of information with entropy H (Bentz and Alikaniotis, 2016;He et al, 2022) in the word w to reflect the importance of that word:…”

Section: Forward Process With Soft-maskingmentioning

confidence: 99%

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

Chen,

Zhang,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffusion LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the categorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our Masked-Diffusion LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency. Code is available at https://github. com/SALT-NLP/Masked_Diffusioin_LM.

show abstract

“…Entropy We also consider measuring the amount of information with entropy H (Bentz and Alikaniotis, 2016;He et al, 2022) in the word w to reflect the importance of that word:…”

Section: Forward Process With Soft-maskingmentioning

confidence: 99%

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

Chen,

Zhang,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…By extracting a distinct inventory of syllables, we establish symbolic representations for each syllable, enabling us to interpret the birdsong as a conventional textual sequence. If the sequence exhibits language-like properties, it is expected to demonstrate similar patterns of word entropy observed in other languages [10]. Jalak Suren (Sturnidae family) is a widely recognized avian species in Indonesia renowned for its melodic and repetitive vocalizations [11].…”

Section: Introductionmentioning

confidence: 98%

Automatic Birdsong Splitting and Syllabic Analysis of Jalak Suren

Prasetiadi,

Saputra

2023

INISTA

View full text Add to dashboard Cite

The study of birdsong has received relatively limited attention in the field of artificial intelligence, despite its long-standing intrigue and the question of whether birds possess a form of language. Previous research has provided evidence suggesting the presence of structurally organized words recognized by birds, such as the strong reactions observed in Japanese tits and Pied babblers when exposed to specific sequences of artificially played calls. Altering the speed of a sequence also influences the birds' responses, further supporting the existence of organized linguistic units in avian vocalizations. In this study, we propose a novel approach for analyzing birdsong by employing automatic syllable segmentation and syllabic similarity analysis. Our focus is on the Jalak Suren species (Sturnus contra), renowned for its melodious song. Through the identification and categorization of distinct syllabic units in birdsong recordings, we investigate the statistical occurrence of these syllables within the sequence of birdsong. Our findings reveal remarkable similarities between the statistical occurrence of syllables in birdsong and those found in human language passages

show abstract

The Spanish subjunctive mood and its English correspondences

Ramón,

Rabadán

2024

LiC

View full text Add to dashboard Cite

Corpus-based contrastive studies have successfully addressed the empirical study of crosslinguistic similarities and differences and may also contribute to understanding complexity across languages. This paper aims at (dis)proving whether the Spanish subjunctive mood shows greater complexity than its English correspondences as translations or sources of the Spanish subjunctive forms. It also explores a trade-off between language levels, i.e., whether higher morphological complexity is linked to syntactic and lexical complexity. The data come from a bidirectional English-Spanish corpus, and the word-alignment-based metric system has been used to quantify morphological complexity. Syntactic and lexical complexity were also investigated and computed using tests of statistical significance. Results corroborate that Spanish presents higher complexity in this verbal area in all three levels, morphological, syntactic, and lexical: the hypothesized trade-off between higher morphological complexity on the one hand, and lower syntactic and lexical complexity on the other is not validated by our data.

show abstract

The word entropy of natural languages

Cited by 4 publications

References 21 publications

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

Automatic Birdsong Splitting and Syllabic Analysis of Jalak Suren

The Spanish subjunctive mood and its English correspondences

Contact Info

Product

Resources

About