Goodness-of-Fit Tests for Large Datasets

Abstract

Nowadays, data analysis in the world of Big Data is connected typically to data mining, descriptive or exploratory statistics, e. g. cluster analysis, classification or regression analysis. Aside these techniques there is a huge area of methods from inferential statistics that are rarely considered in connection with Big Data. Nevertheless, inferential methods are also of use for Big Data analysis, especially for quantifying uncertainty.The article at hand will provide some insights to methodological and technical issues referring inferential methods in the Big Data area in order to bring together Big Data and inferential statistics, as it comes along with its difficulties. We present an approach that allows testing goodness-of-fit without model assumptions and relying on the empirical distribution. Especially, the method is able to utilize information from large datasets. Thereby, the approach is based on a clear theoretical background. We concentrate on the widely-used Kolmogorov-Smirnov test that is applied for testing goodness-of-fit in statistics. Our approach can be parallelized easily, which makes it applicable to distributed datasets particularly on a compute cluster.By this contribution, we turn to an audience that is interested in the technical and methodological backgrounds while implementing especially inferential statistical methods with Big Data tools as e. g. Spark.

ALMA-LEGUS. II. The Influence of Subgalactic Environments on Molecular Cloud Properties

“…As thoroughly discussed in Lazariv & Lehmann (2018), as the sample size increases, the discerning power of KS tests increases. However, KS tests cannot take into account error in the measurements, and so it is possible for a test to become overpowered.…”

“…As in Paper I, we quantify these differences in property distributions using bootstrapped two-sample Kolmogorov-Smirnov (KS) tests and Anderson-Darling (AD) tests. Using the full sample size of the galaxies results in an overpowered statistic as described in Lazariv & Lehmann (2018), where the tests detected every small variation in the distribution, even those that were well below the measurement errors. Since these overpowered tests are unreliable, we instead perform a bootstrapped version, taking 1000 random subsamples and reporting the average p-value of the tests performed on the subsamples.…”

ALMA-LEGUS. I. The Influence of Galaxy Morphology on Molecular Cloud Properties

“…This is likely an overrepresentation of how different the distributions truly are though because the sample sizes of the clouds and clusters for the galaxies are so large. As discussed in Lazariv & Lehmann (2018), as the sample size becomes larger, K-S and AD tests have increasingly higher power to discern small differences in the distributions. However, these tests do not take into account the errors in the parameters, and so at large sample sizes, these tests can discern differences that are smaller than the errors in the measurements, which we consider unreliable.…”

Finding the LMA needle in the wheat proteome haystack

“…To assess whether our LC-MS1 datasets following the correction and z-transformation steps was normally distributed, we plotted the data as histogram and boxplot. We further performed the nonparametric one-sample Kolmogorov-Smirnov (K-S) test (85) well suited to analysing big data (86). Both histogram and boxplot of the corrected data were asymmetrical with most values being on the low range (Supplementary Figure 6A-B), which revealed that this dataset was not normally distributed.…”