Type I error and the number of iterations in Monte Carlo studies of robustness

Abstract

A recent survey of simulation studies concluded that an overwhelming majority of papers do not report a rationale for the decision regarding the number of Monte Carlo iterations. A surprisingly large number of reports do not contain a justifiable definition of robustness and many studies are conducted with an insullicient number of iterations to achieve satisfactory statistical conclusion validity. The implication is that we do not follow our own advice regarding the management of Type I and Type 11 errors when conducting Monte Carlo experiments. This paper reports a straightforward application of a well-known procedure for the purpose of objectively determining the exact number of iterations necessary to confidently detect departures from robustness in Monte Carlo results. A table of the number of iterations necessary to detect departures from a series of nominal Type I error rates is included.

Comparación de los procedimientos de Fleishman y Ramberg et al. para generar datos no normales en estudios de simulación

“…In these cases the lowest deviation is produced between 13000 and 15000 simulations, where the kurtosis deviation takes a value of 0.20. Although these results cannot be directly compared with other studies, as this is the first report to compare the procedures of Fleishman (1978) and Ramberg et al (1979) in terms of the number of simulations required to generate non-normal data, our findings are partially consistent with previous research stating that the quality of simulation studies can be increased by using 10000 simulations (Bang et al, 1998;Rasch & Guiard, 2004;Robey & Barcikowski, 1992).…”

Statistical criteria for parallel tests: A comparison of accuracy and power

“…2b), the Student-Pitman-Morgan test has deflated empirical type I error rates at 8.70 %, 4.34 %, and 0.80 %, respectively, for nominal significance levels of 10 %, 5 %, and 1 %. It should nevertheless be pointed out that test sizes that are within 20 % of the nominal size (i.e., actual sizes no lower than 8 %, 4 %, or 0.8 %, respectively, for nominal test sizes of 10 %, 5 %, or 1 %) are slightly above J. V. Bradley's (1978) "fairly stringent" criterion for robustness (±10 %), but they are acceptable by Robey and Barcikowski's (1992) …”

“…If test scores were real-valued and unbounded, BradleyBlackwood and Student-Pitman-Morgan tests would maintain their nominal sizes for n > 50 (the smallest sample size used in our simulations, which is usually exceeded in studies of parallelism), whether scores were symmetrically distributed (from normal or uniform distributions) or asymmetrically distributed (from folded normal distributions) within the limits typically observed in empirical test score distributions. With (inescapably) bounded test scores, both tests become slightly and equally conservative, although their actual test size remains well within 20 % of the nominal size, something that is usually regarded as acceptable (García-Pérez & Núñez-Antón, 2009;Robey & Barcikowski, 1992;Serlin, 2000;Serlin & Harwell, 2004). Because the conservatism of both tests depends on the extent to which the distribution of observed scores is curtailed by the hard bounds of minimal and maximal scores, cautious .8 1 Fig.…”

Determining The Correct Number Of Components To Extract From A Principal Components Analysis: A Monte Carlo Study Of The Accuracy Of The Scree Plot

“…They use computer-assisted simulations to provide evidence for problems that cannot be solved mathematically. Robey and Barcikowski (1992) stated that in Monte Carlo simulations, the values of a statistic are observed in many samples drawn from a defined population.…”