Statistical Debugging for Simulations

Abstract

Predictions from simulations have entered the mainstream of public policy and decision-making practices. Unfortunately, methods for gaining insight into faulty simulations outputs have not kept pace. Ideally, an insight gathering method would automatically identify the cause of a faulty output and explain to the simulation developer how to correct it. In the field of software engineering, this challenge has been addressed for general-purpose software through statistical debuggers. We present two research contributions, elastic predicates and many-valued labeling functions, that enable debuggers designed for general-purpose software to become more effective for simulations employing random variates and continuous numbers. Elastic predicates address deficiencies of existing debuggers related to continuous numbers, whereas manyvalued labeling functions support the use of random variates. When used in combinations, these contributions allow a simulation developer tasked with localizing the program statement causing the faulty simulation output to examine 40% fewer statements than the leading alternatives. Our evaluation shows that elastic predicates and many-valued labeling functions maintain their ability to reduce the number of program statements that need to be examined under the imperfect conditions that developers experience in practice.

Aging Out of Crime: Exploring the Relationship Between Age and Crime with Agent-Based Modeling

“…As an additional form of validation we explore those simulation parameterizations that yield an increase in the number of criminals after 100 time steps using exploration techniques based on statistical debugging described in Gore et al (2015), Diallo et al (2016), and Gore, Lynch, and Havak (2016). In each parameterization that increases the number of criminals in our extended model, the life stage where the maximum number of resources is spent is never: (a) criminals, or (b) Life Stage 1.…”

A content analysis-based approach to explore simulation verification and identify its current challenges

“…Verification costs increase as a result of increasing system autonomy, complexity, and abilities to assess their own status [77]. These characteristics reflect challenges pertaining to parallel execution, large amounts of simulated data [15,78], building and running models in the cloud [38,39,60,79,80], and tracing the occurrences of errors to their sources [61,81,82].…”

“…As a result, prerequisite statistical, mathematical, and simulation knowledge requirements are further increased. Statistical debugging using elastic predicates and many-valued labeling functions has been developed for the exploration of simulation using software engineering principles and directly accounts for sample sizes when making verification determinations [82]. Statistical debugging delves into complex simulation interactions without requiring a formal mathematical model specification to identify and isolate locations of potential errors [61,82,111].…”

Emergent behavior identification within an agent-based model of the Ballistic Missile Defense System using statistical debugging

“…Simulation predicates with a high suspiciousness score frequently occur when the simulation exhibits the behavior, whereas predicates with a low suspiciousness score frequently occur when the simulation does not exhibit the behavior. 19,20 An example helps elucidate how statistical debugging can be applied to localize a condition causing unsuccessful outputs. The example involves an agent-based simulation that studies the role of restaurant choices on the spread of obesity in an area.…”