Number of Replications Required in Monte Carlo Simulation Studies: A Synthesis of Four Studies

Mundform, Daniel J.; Schaffer, Jay; Kim, Myoung-Jin; Shaw, Dale G.; Thongteeraparp, Ampai; Supawan, Pornsin

doi:10.22237/jmasm/1304222580

Cited by 72 publications

(54 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, the larger the number of iterations, the more the overall results of the simulation are stable. According to [41], 10,000 iterations is a suitable number for simulations of this type. For each iteration, the proposed method was applied, and the test's behavior was analyzed.…”

Section: Resultsmentioning

confidence: 99%

A New Method for Positional Accuracy Control for Non-Normal Errors Applied to Airborne Laser Scanner Data

et al. 2019

View full text Add to dashboard Cite

A new statistical method for the quality control of the positional accuracy, useful in a wide range of data sets, is proposed and its use is illustrated through its application to airborne laser scanner (ALS) data. The quality control method is based on the use of a multinomial distribution that categorizes cases of errors according to metric tolerances. The use of the multinomial distribution is a very novel and powerful approach to the problem of evaluating positional accuracy, since it allows for eliminating the need for a parametric model for positional errors. Three different study cases based on ALS data (infrastructure, urban, and natural cases) that contain non-normal errors were used. Three positional accuracy controls with different tolerances were developed. In two of the control cases, the tolerances were defined by a Gaussian model, and in the third control case, the tolerances were defined from the quantiles of the observed error distribution. The analysis of the test results based on the type I and type II errors show that the method is able to control the positional accuracy of freely distributed data.

show abstract

Section: Resultsmentioning

confidence: 99%

A New Method for Positional Accuracy Control for Non-Normal Errors Applied to Airborne Laser Scanner Data

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Several conclusions can be derived from these results and figures. First, as implied by Mundfrom et al (2011), the size of CI half-widths may reach an acceptable level (i.e., 0.005) somewhere below 10,000 replications, under some conditions, as exemplified in Figure 2(b). The challenge is to find an appropriate number of replications to balance the internal validity and external validity of MC experiments when varying conditions require differing numbers of replications.…”

Section: Discussionmentioning

confidence: 96%

“…In our quick review, only four articles provided a rationale: two based loosely on Robey and Barcikowski and two that justified their number of replications with phrases like "to ensure stable results" (curiously, one used 1,000 replications to "ensure" stability and the other used 10,000). Based on a research synthesis, Mundfrom et al (2011) suggested that approximately 5,000-8,000 appeared sufficient to produce stable Monte Carlo results, depending on the purpose of the research. Harwell, Stone, Hsu, and Kirisci (1996) concluded that "clearly, more research in this area is needed before there is a definitive answer concerning the number of replications that should be used, given the purpose and conditions of a particular MC study" (p. 112).…”

Section: Recommendations From the Literaturementioning

confidence: 99%

A Precision-Based and Adaptive Approach to Number of Replications for Monte Carlo Studies of Robustness and Power

Brooks¹,

Diaz²,

Johanson³

2017

GLMJ

View full text Add to dashboard Cite

MC) researchers must determine how many replications or repeated samples to draw for each condition under investigation. MC experiments performed with too few replications may produce idiosyncratic results, but too many replications may be inefficient. The purpose of this paper is to examine the number of replications needed in MC experiments designed to investigate robustness and statistical power. A precision-based method for determining an appropriate number of replications, uniquely combined here with robustness criteria, is recommended. Using analytical and meta-Monte Carlo methods, implications of this precision-based method are considered and tables for the recommended number of replications based on the method are provided. Further, recommendations are made to enhance both the accuracy and consistency of MC studies of robustness and power using an adaptive, continuous criterion comparison method of programming combined with the precision-based approach. Ultimately, we show that tables provided here can also be used when MC researchers desire an appropriate number of replications for estimating both proportion parameters (e.g., Type I error, statistical power) and nonproportion parameters (e.g., means, regression coefficients).onte Carlo (MC) methods are used in statistics for various purposes, especially when analytical solutions or closed formulas are not possible or not easy (Mooney, 1997). In robustness studies of Type I error rates, MC researchers use computer simulation to draw many repeated samples of pseudorandom data from population distributions with known parameters such that the null hypothesis is true; therefore, all rejections of the null hypothesis are Type I errors. After many repeated samples, the proportion of rejections (π) estimates the actual probability of a Type I error under the studied conditions (e.g., violations of the statistic's assumptions). Robustness is determined by how well π estimates the nominal Type I error rate, α. In MC studies of power, conditions are set such that null hypotheses are known to be false, where the proportion of the repeated samples in which the null hypothesis is correctly rejected is used to estimate statistical power. If desired, Type II error is then generally inferred as the complement to power (i.e., Type II error = 1power). In this case, robustness can then be determined by how well 1 -π estimates the nominal Type II error rate, β.MC researchers must determine how many repeated samples (also called replications, trials, or iterations) to draw for each condition under investigation. MC experiments performed with too few replications may produce inaccurate, unstable, and idiosyncratic results, but too many replications may be inefficient (Hutchinson & Bandalos, 1997). That is, more replications result in more statistical power to detect departures from theory and more precision to estimate parameters, but there are diminishing returns as the number of replications increases.The purpose of this paper is to examine the number of replications needed ...

show abstract

“…Different approaches can be used for calculating the run length (RL) values. In our study, we used Monte Carlo Simulation approach to obtain the values of all estimators under simple random sampling technique using R software . We get an RL value by simulating 100 000 values of a particular estimator through Monte Carlo simulation.…”

Section: Performance Measuresmentioning

confidence: 99%

On a class of mixed EWMA‐CUSUM median control charts for process monitoring

Hussain

Wang

Ahmad

et al. 2020

Quality & Reliability Eng

View full text Add to dashboard Cite

Monitoring of any manufacturing, production, or industrial process can be controlled and improved by removing these special cause of variations using control charts. Shewhart‐type control charts are effective to control a large amount of special variations, whereas, cumulative sum (CUSUM) and exponentially weighted moving average (EWMA) charts detect small and moderate variations efficiently in the process parameters. Monitoring of location parameter can be done with mean control charts under the assumption that the parameters are known or correctly estimated from in‐control samples and data are free from outliers (but in practice data occasionally have outliers). In this study, we have proposed generalized mixed EWMA‐CUSUM median control charts structures for known and unknown parameters based on auxiliary variables for detecting shifts in process location parameter. The proposed charts are compared with the corresponding charts for the mean, based on contaminated and uncontaminated data. Different performance measures are used to evaluate the performance of proposed control charts and revealed through results that the median‐based charts are more sensitive to detect a shift in process location parameter in the presence of outliers. An illustrative example using real data is also shown for practical consideration.

show abstract

Number of Replications Required in Monte Carlo Simulation Studies: A Synthesis of Four Studies

Cited by 72 publications

References 23 publications

A New Method for Positional Accuracy Control for Non-Normal Errors Applied to Airborne Laser Scanner Data

A New Method for Positional Accuracy Control for Non-Normal Errors Applied to Airborne Laser Scanner Data

A Precision-Based and Adaptive Approach to Number of Replications for Monte Carlo Studies of Robustness and Power

On a class of mixed EWMA‐CUSUM median control charts for process monitoring

Contact Info

Product

Resources

About