Use of thep-valueas a size-dependent function to address practical differences when analyzing large datasets

Abstract

The p-value is routinely compared with a certain threshold, commonly set to 0.05, to assess statistical null hypotheses. This threshold is easily reachable by either a single p-value or its distribution whenever a large enough dataset is available. We prove that the p-value can be alternatively modeled as a continuous exponential function. The function's decay can be used to analyze the data, assess the null hypothesis, and determine the minimum data-size needed to reject it. An in-depth study of the model in three different experimental datasets reflects the large scope of this approach in common data analysis and decisionmaking processes. In Fig. 1a, different randomly generated normal distributions are compared using the Mann-Whitney U statistical test [11] to illustrate the decrease of the function ( ) with the sample size. The use of the Student's t-test was avoided as it is well known that the p-value associated to the t-statistic has an exponential decay [13]. Technical details about the convergence of the function ( ) and evidence about Eq. 1 holding for any statistical test are given in the Online Methods.Note that the p-value curve, the function ( ), is used to compare pairs of experimental conditions; therefore, ( ) is computed as the exponential fit to the probability value of multiple sample comparisons.Hence, the parameters and in Eq. 1 correspond to those defining the exponential fit ( ). We use the Monte Carlo cross-validation (MCCV) [16] as the sampling strategy: two subsets of size n (one from each of the groups to be compared) are randomly sampled and compared with a statistical test. The resulting p-value is stored and the procedure is repeated many times. At the end of the procedure, a large set of -dependent p-values is obtained and the exponential function in Eq. 1 can be fit.Similarly to any exponential function, ( ) converges to zero. The faster the function converges, the more robust the significance. When normal distributions of standard deviation one and mean value in the range [0, 3] are compared, we see that the higher the difference among experimental conditions, the faster the decay of the exponential function that approximates ( ) ( Fig. 1b). We observe that the parameters and (Eq. 1) increase proportionally with the mean value of the distribution compared with (0, 1) ( Fig. 1b). With this new idea in mind, a robust decision index, , , can be mathematically defined (Eq. 10 in the Online Methods). Note that subscripts (statistical significant threshold) and (regularization parameter) are omitted from now on.Instead of comparing a single p-value with the ideal statistical significance threshold (i.e., = 0.05 for a 95% of statistical significance), a distance (Eq. 9 in the Online Methods) is defined to compare the function ( ) with for all values. measures the difference between the areas under the constant function at level and the area under the curve ( ) (Fig. 1c). The distance is then used to obtain the binary index that indicates whether ( ) and the constant are far from each ot...

Grouped star formation: converting sink particles to stars in hydrodynamical simulations

“…The velocity dispersion of L20 is fitted with a Gaussian distribution with mean 6.2 km s −1 and standard deviation of 2.7 km s −1 (black dashed line). As large sample sizes cause bias towards rejecting the null hypothesis for equal distributions, KS tests are not suitable for hypothesis testing here (G ómez-de Mariscal et al 2021 ). None the less, all velocity distributions are visually Gaussian-like and similar to that of L20, regardless of the grouping parameters, suggesting that the velocity distribution of the larger scale models is relatively independent of the details of the star formation prescription.…”