MATLAB tool for probability density assessment and nonparametric estimation

Farmer, Jenny; Jacobs, Donald J.

doi:10.1016/j.softx.2022.101017

Cited by 9 publications

(10 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…а також доволі просто виразити квантилі порядку Q q на сітці Q квантилів, розв'язуючи рівняння, яке випливає з виразу (11):…”

Section: обґрунтування вибору показників та критеріїв незалежності рр...unclassified

“…У доволі простих тестах, окрім лінійних залежностей (кореляцій), аналізують і моментні статистики ПВЧ вищих порядків [10,11]. Найчастіше тестування ПВЧ розуміють згідно із парадигмою " достатньої випадковості" [12], наприклад, за допомогою системи тестів американського стандарту NIST 800-22, 1a [13].…”

unclassified

“…Функція розподілення(11) та щільність розподілення(12) Простота, так би мовити, " математична елегантність" виразів(11) та (12) дає змогу елементарно визначити математичне очікування ( S M ), дисперсію ( S D ) та стандартне відхилення ( S s ) для цих функцій:…”

unclassified

See 2 more Smart Citations

Sum criteria for the task of testing the independence of random numbers sequences

Odegov,

Babich,

Bagachuk

et al. 2023

ICTEE

View full text Add to dashboard Cite

Random and pseudo-random number generators (RNGs) were initially used to solve numerical integration problems (the Monte Carlo method). Currently, the RNGs are used in cryptography and simulation modeling. The latter one typically uses RNGs based on computer algorithms and programs. This article presents a method aimed at testing the independence of random numbers sequences (RNSs). The method is based on the sums properties of independent random variables. Algorithms based on this method operate fast. Here not only the instant statistics including correlation coefficients are analyzed, but also the properties of empirical functions of RNSs distributed sums. In this article, the analysis is limited only to the case of uniformly distributed RNSs. The calculations performed prove the high selective efficiency of the proposed criteria, which allows to reliably distinguish between dependent and independent RNSs. Due to the high operation speed, the proposed algorithms and criteria can be used for testing very long RNSs (especially in Big Data tasks).

show abstract

Section: обґрунтування вибору показників та критеріїв незалежності рр...unclassified

unclassified

See 1 more Smart Citation

Sum criteria for the task of testing the independence of random numbers sequences

Odegov,

Babich,

Bagachuk

et al. 2023

ICTEE

View full text Add to dashboard Cite

show abstract

“…estimatePDF estimatePDF provides an R interface for nonparametric density estimation based on a novel method providing an alternative to traditional KDE implementations. Details of this approach, based on the principle of maximum entropy (Wu, 1997), were published previously and have been shown to produce more accurate estimates than KDE in most cases (Farmer and Jacobs, 2018;Farmer et al, 2019;Puchert et al, 2021;Farmer and Jacobs, 2022). For optimal performance and flexibility with other applications, this functionality is performed within a set of C++ classes and is not the focus of this paper, but a brief summary is provided for insight into the estimatePDF interface.…”

Section: Gettarget and Plotbetamentioning

confidence: 99%

PDFEstimator: An R Package for Density Estimation and Analysis

Farmer¹,

Jacobs²

2022

The R Journal

View full text Add to dashboard Cite

This article presents PDFEstimator, an R package for nonparametric probability density estimation and analysis, as both a practical enhancement and alternative to kernel-based estimators. PDFEstimator creates fast, highly accurate, data-driven probability density estimates for continuous random data through an intuitive interface. Excellent results are obtained for a diverse set of data distributions ranging from 10 to 10 6 samples when invoked with default parameter definitions in the absence of user directives. Additionally, the package contains methods for assessing the quality of any estimate, including robust plotting functions for detailed visualization and trouble-shooting. Usage of PDFEstimator is illustrated through a variety of examples, including comparisons to several kernel density methods.

show abstract

“…In specific applications this diffusion approach delivered convincing results (e.g., Botev et al, 2010;Deniz et al, 2011;Qin and Xiao, 2018). However, it tends to resolve too many details or overfit the data in others (e.g., Ma et al, 2019;Chaudhuri and Marron, 2000;Farmer and Jacobs, 2022). One main benefit of the diffusion KDE is that it provides a series of PDF estimates for a sequence of bandwidths by default (Chaudhuri and Marron, 2000).…”

Section: Introductionmentioning

confidence: 99%

A diffusion-based kernel density estimator (diffKDE, version 1) with optimal bandwidth approximation for the analysis of data in geoscience and ecological research

Verwega

Schartau²,

Somes³

et al. 2023

Preprint

View full text Add to dashboard Cite

Abstract. Probability density functions (PDFs) comprise basic information about the variability of observed or simulated variables within a system of interest. In geoscience data distributions are often expressed by a parametric estimation of their PDF, such as e.g. a Gaussian distribution. At present there is a growing attention towards the analysis of non-parametric estimation of PDFs, where no prior assumptions about the type of PDF are required. A common tool for such non-parametric estimation is a kernel density estimator (KDE). Existing KDEs are valuable but incomplete, because of the difficulty of specifying optimal bandwidths for the individual kernels. A diffusion-based KDE provides a useful approach to mitigate the difficulty in identifying bandwidths that resolve desired details of multi-modal data while being insensitive to noise. Therefore we designed and developed a new implementation of a diffusion-based KDE as an open source Python tool. We tested our implementation on artificial and real marine biogeochemical data individually and against other popular KDEs. Our estimator is able to detect relevant multiple modes and resolve boundary close data while suppressing details induced by noise and individual outliers. The convergence rate is comparable to the Gaussian estimator, but with a generally smaller error, most notably for small data sets with up to around 5000 data points. We exemplify and discuss the general applicability of such KDEs for data-model comparison in geoscience, in particular for sparse data. We also provide an example for how our approach can be efficiently utilized for the derivation of plankton size spectra in ecological research.

show abstract

MATLAB tool for probability density assessment and nonparametric estimation

Cited by 9 publications

References 19 publications

Sum criteria for the task of testing the independence of random numbers sequences

Sum criteria for the task of testing the independence of random numbers sequences

PDFEstimator: An R Package for Density Estimation and Analysis

A diffusion-based kernel density estimator (diffKDE, version 1) with optimal bandwidth approximation for the analysis of data in geoscience and ecological research

Contact Info

Product

Resources

About