Estimating the Entropy of Linguistic Distributions

Arora, Aryaman; Meister, Clara; Cotterell, Ryan

doi:10.18653/v1/2022.acl-short.20

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We note that some recent work (Meister et al, 2021;Arora et al, 2022) has also described estimating statistics of random variables induced by language models. This involved sampling without replacement and using importance weighting, but they seem to surpass Monte-Carlo estimates only with many samples and peaky distributions.…”

Section: Likelihoodmentioning

confidence: 97%

Selectively Answering Ambiguous Questions

Cole,

Zhang,

Gillick

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Trustworthy language models should abstain from answering questions when they do not know the answer. However, the answer to a question can be unknown for a variety of reasons. Prior research has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown. But the answer to a question can also be unclear due to uncertainty of the questioner's intent or context. We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous. In this setting, we find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs, rather than the model's likelihood or self-verification as used in prior work. We find this to be the case across different types of uncertainty and model scales, and with or without instruction tuning. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous questions.

show abstract

Section: Likelihoodmentioning

confidence: 97%

Selectively Answering Ambiguous Questions

Cole,

Zhang,

Gillick

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Precisely, the previous literature has largely dealt with entropy estimators proposed for sequences of i.i.d. random variables [ 16 , 18 , 19 , 20 , 21 ]. However, it is not clear that real data arising from experimental observation can be described with i.i.d.…”

Section: Introductionmentioning

confidence: 99%

Entropy Estimators for Markovian Sequences: A Comparative Analysis

De Gregorio,

Sánchez,

Toral

2024

Entropy

View full text Add to dashboard Cite

Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performances vary depending on the system being studied and the available data size. In this work, we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation, and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields.

show abstract

Learning-based Property Estimation with Polynomials

Li,

Lei,

Wang

et al. 2024

Proc. ACM Manag. Data

View full text Add to dashboard Cite

The problem of estimating data properties using sampling frequency histograms has attracted extensive interest in the area of databases. The properties include the number of distinct values (NDV), entropy, and so on. In the field of databases, property estimation is fundamental to complex applications. For example, NDV estimation is the foundation of query optimization, and entropy estimation is the foundation of data compression. Among them, methods originating from statistics exhibit desirable theoretical guarantees but rely on specific assumptions about the distribution of data, resulting in poor performance in real-world applications. Learning-based methods, which use information from training data, are adaptable in the real world but often lack theoretical guarantees or explainability. In addition, a unified framework for estimating these frequency-based estimators with machine learning is lacking. Given the aforementioned challenges, it is natural to wonder if a unified framework with theoretical guarantees can be established for property estimation. The recent literature has presented theoretical studies that propose estimation frameworks based on polynomials. These studies also prove estimation errors with respect to the sample size. Motivated by the above polynomial estimation framework, we propose a learning-based estimation framework with polynomial approximation, which aims to learn the coefficients of the polynomial, providing theoretical guarantees to the learning framework. Through comprehensive experiments on both synthetic and real-world datasets for estimating various data properties like NDV, entropy, and power sum, our results show the superiority of our algorithms over previous estimators.

show abstract

Estimating the Entropy of Linguistic Distributions

Cited by 3 publications

References 5 publications

Selectively Answering Ambiguous Questions

Selectively Answering Ambiguous Questions

Entropy Estimators for Markovian Sequences: A Comparative Analysis

Learning-based Property Estimation with Polynomials

Contact Info

Product

Resources

About