Systematic Investigation of Variability due to Random Simulation Error in an Activity-Based Microsimulation Forecasting Model

Abstract

A key difference between stochastic microsimulation models and more traditional forms of travel demand forecasting models is that micro-simulation-based forecasts change each time the sequence of random numbers used to simulate choices is varied. To address practitioners’ concerns about this variation, a common approach is to run the microsimulation model several times and average the results. The question then becomes: What is the minimum number of runs required to reach a true average state for a given set of model results? This issue was investigated by means of a systematic experiment with the San Francisco model, a microsimulation model system used in actual planning applications since 2000. The system contains models of vehicle availability, day pattern choice, tour time-of-day choice, destination choice, and mode choice. To investigate the variability of the forecasts of this system due to random simulation error, the model system was run 100 times, each time changing only the sequence of random numbers used to simulate individual choices from the logit model probabilities. The extent of random variability in the model results is reported as a function of two factors: ( a) the type of model (vehicle availability, tour generation, destination choice, or mode choice); and ( b) the level of geographic detail—transit at the analysis zone level, neighborhood level, or countywide level. For each combination of these factors, it is shown graphically how quickly the mean values of key output variables converge toward a stable value as the number of simulation runs increases.

A multidimensional mixed ordered-response model for analyzing weekend activity participation

“…Specifically, several operational analytic frameworks within the activity analysis paradigm have been formulated, and some metropolitan areas have even implemented these frameworks (Waddell et al, 2002 andCastiglione et al, 2003).…”

Making Household Microsimulation of Travel and Activities Accessible to Planners

“…The microsimulation run time increases linearly with the size of the synthetic population. As elaborated further below, the size of the synthetic population and the number of runs that are required is based on the statistic of interest for the analysis (21). For urban-wide statistics (for example, total VMT), one run using 20% population appears to be sufficient.…”

“…The advantage of sampling error (as opposed to aggregation error) is that one can estimate the size of the error and generate confidence intervals. (21) presented a thorough analysis of sampling error resulting from microsimulation runs of the San Francisco model. They used a full synthetic population, and show that the magnitude of the sampling error varies based on the characteristics of the statistic of interest.…”

Investigating micro-simulation error in activity-based travel demand forecasting: a case study of the FEATHERS framework

“…The question then becomes: what is the minimum number of runs required to reach a stable result (i.e., with a certain level of confidence that the obtained average value can only vary within an acceptable interval)? In this respect, several relevant studies have Benekohal and Abu-Lebdeh (1994), Hale (1997), Veldhuisen et al (2000b, Esser andNagel (2001), Vovsha et al (2002), Castiglione et al (2003), Ziems et al…”

“…(2011), Horni et al (2011), and Cools et al (2011). In particular, Castiglione et al (2003) investigated the extent of random variability in the San Francisco model (a micro-simulation model system) by running the model 100 times at three levels of geographic detail, namely zone level, neighborhood level, and county-wide level. The analysis was then conducted by showing how quickly the mean values of output variables such as the number of trips per person converge towards the final mean value (after 100 runs) as the number of simulation runs increases.…”