Evaluation of Software for Propagating Uncertainty Through Risk Assessment Models

Abstract

Quantitative uncertainty analysis has become a common component of risk assessments. In risk assessment models, the most robust method for propagating uncertainty is Monte Carlo simulation. Many software packages available today offer Monte Carlo capabilities while requiring minimal learning time, computational time, and/or computer memory. This paper presents an evaluation of six software packages in the context of risk assessment: Crystal Ball, @Risk, Analytica, Stella II, PRISM, and Susa-PC. Crystal Ball and @Risk are spreadsheet-based programs; Analytica and Stella II are multi-level, influencediagram based programs designed for the construction of complex models; PRISM and Susa-PC are both public-domain programs designed for incorporating uncertainty and sensitivity into any model written in Fortran. Each software package was evaluated on the basis of five criteria, with each criterion having several sub-criteria. A 'User Preferences Table' was also developed for an additional comparison of the software packages. The evaluations were based on nine weeks of experimentation with the software packages including use of the associated user manuals and test of the software through the use of example problems. The results of these evaluations indicate that Stella II has the most extensive modeling capabilities and can handle linear differential equations. Crystal Ball has the best input scheme for entering uncertain parameters and the best reference materials. @Risk offers a slightly better standard output scheme and requires a little less learning time. Susa-PC has the most Metzger et al.options for detailed statistical analysis of the results, such as multiple options for a sensitivity analysis and sophisticated options for inputting correlations. Analytica is a versatile, menu-and graphics-driven package, while PRISM is a more specialized and less user friendly program. When choosing between software packages for uncertainty and sensitivity analysis, the choice largely depends on the specifics of the problem being modeled. However, for risk assessment problems that can be implemented on a spreadsheet, Crystal Ball is recommended because it offers the best input options, a good output scheme, adequate uncertainty and sensitivity analysis, superior reference materials, and an intuitive spreadsheet basis while requiring very little memory.

Dependence in probabilistic modeling, Dempster-Shafer theory, and probability bounds analysis.

“…These constraints are rather complicated, but can be summarized by saying that a correlation matrix must be positive semi-definite (see Section 3.6.1). Early versions of the software package @Risk (Palisade Corporation 1996;Salmento et al 1989;Barton 1989;Metzger et al 1998) did not account for this constraint, and consequently would have produced nonsensical results whenever users would specify an infeasible set of correlations.…”

“…comm. ;Decisioneering 1996;Burmaster and Udell 1990;Metzger et al 1998) and is probably the most widely used method for inducing correlations in Monte Carlo simulations. However, like the "Lurie and Goldberg (1994) described an iterative approach for obtaining a desired pattern of Pearson correlations matching specified marginal distributions, but it is essentially a trial-anderror approach that can be computationally intensive.…”