A comparison of experimental designs in the development of a neural network simulation metamodel

Alam, M Fasihul; McNaught, Ken R.; Ringrose, Trevor J.

doi:10.1016/j.simpat.2003.10.006

Cited by 96 publications

(50 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Clarke et al (2003) compare low-order polynomials, radial basis functions, Kriging, splines, and Support Vector Regression. Alam et al (2003) found that LHS gives the best neural-net metamodels. Comparison of screening designs has hardly been done; see Kleijnen et al (2003 a, b).…”

Section: Conclusion and Further Researchmentioning

confidence: 97%

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

Kleijnen

2004

SSRN Journal

View full text Add to dashboard Cite

Section: Conclusion and Further Researchmentioning

confidence: 97%

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

Kleijnen

2004

SSRN Journal

View full text Add to dashboard Cite

“…To the author's knowledge of the author, no actual evaluation of their sampling efficiency relative to normal Latin hypercube sampling has been made to date. Several authors do consider maximin and uniform designs when evaluating the performance of metamodels [34], [55], [56], but the influences of the metamodelling method and the sampling strategy cannot be separated. In this paper hence, such evaluation of the sampling efficiency of basic random versus standard, maximin and uniform Latin hypercube sampling designs is executed.…”

Section: Sampling Efficiencymentioning

confidence: 99%

Monte-Carlo based uncertainty analysis: Sampling efficiency and sampling convergence

Janssen

2013

Reliability Engineering & System Safety

313

131

View full text Add to dashboard Cite

Monte Carlo analysis has become nearly ubiquitous since its introduction, now over 65 years ago. It is an important tool in many assessments of the reliability and robustness of systems, structures or solutions. As the deterministic core simulation can be lengthy, the computational costs of Monte Carlo can be a limiting factor. To reduce that computational expense as much as possible, sampling efficiency and convergence for Monte Carlo are investigated in this paper. The first section shows that non-collapsing spacefilling sampling strategies, illustrated here with the maximin and uniform Latin hypercube designs, highly enhance the sampling efficiency, and render a desired level of accuracy of the outcomes attainable with far lesser runs. In the second section it is demonstrated that standard sampling statistics are inapplicable for Latin hypercube strategies. A sample-splitting approach is put forward, which in combination with a replicated Latin hypercube sampling allows assessing the accuracy of Monte Carlo outcomes. The assessment in turn permits halting the Monte Carlo simulation when the desired levels of accuracy are reached. Both measures form fairly noncomplex upgrades of the current state-of-the-art in Monte-Carlo based uncertainty analysis but give a substantial further progress with respect to its applicability. KEYWORDSMonte Carlo, uncertainty analysis, space-filling Latin hypercube, sampling efficiency, sampling convergence, sample-splitting HIGHLIGHTS  Monte Carlo is virtually universal, but computational expense is important barrier  A good sampling strategy and convergence assessment will improve applicability  Space-filling Latin hypercube designs are most efficient, should be generally used  Sample-splitting on replicated Latin hypercube designs allows assessing accuracy  Both measures are undemanding upgrades, but substantially boost the applicability Postprint: Janssen H. 2013. Monte-Carlo based uncertainty analysis: Sampling efficiency and sampling convergence, Reliability Engineering & System Safety,

show abstract

“…They are particularly useful for fitting nonparametric models, such as locally weighted regressions. These designs, especially Latin hypercube sampling (LHS), have been applied when fitting Kriging models (see §4) and neural networks (Alam et al 2004). Detection of thresholds is discussed by Watson and Barnes (1995), who propose a sequential design procedure.…”

Section: Space Filling and Bias Protectionmentioning

confidence: 99%

State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments

Kleijnen

Sanchez

Lucas

et al. 2005

INFORMS Journal on Computing

386

212

View full text Add to dashboard Cite

Many simulation practitioners can get more from their analyses by using the statistical theory on design of experiments (DOE) developed specifically for exploring computer models. We discuss a toolkit of designs for simulators with limited DOE expertise who want to select a design and an appropriate analysis for their experiments. Furthermore, we provide a research agenda listing problems in the design of simulation experiments-as opposed to real-world experiments-that require more investigation. We consider three types of practical problems: (1) developing a basic understanding of a particular simulation model or system, (2) finding robust decisions or policies as opposed to so-called optimal solutions, and (3) comparing the merits of various decisions or policies. Our discussion emphasizes aspects that are typical for simulation, such as having many more factors than in real-world experiments, and the sequential nature of the data collection. Because the same problem type may be addressed through different design types, we discuss quality attributes of designs, such as the ease of design construction, the flexibility for analysis, and efficiency considerations. Moreover, the selection of the design type depends on the metamodel (response surface) that the analysts tentatively assume; for example, complicated metamodels require more simulation runs. We present several procedures to validate the metamodel estimated from a specific design, and we summarize a case study illustrating several of our major themes. We conclude with a discussion of areas that merit more work to achieve the potential benefits-either via new research or incorporation into standard simulation or statistical packages.

show abstract

A comparison of experimental designs in the development of a neural network simulation metamodel

Cited by 96 publications

References 16 publications

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

Monte-Carlo based uncertainty analysis: Sampling efficiency and sampling convergence

State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments

Contact Info

Product

Resources

About