Holistically evaluating agent-based social systems models: a case study

Abstract

The philosophical perspectives on model evaluation can be broadly classified into reductionist/logical positivist and relativist/holistic. In this paper, we outline some of our past efforts in, and challenges faced during, evaluating models of social systems with cognitively detailed agents. Owing to richness in the model, we argue that the holistic approach and consequent continuous improvement are essential to evaluating complex social system models such as these. A social system built primarily of cognitively detailed agents can provide multiple levels of correspondence, both at observable and abstract aggregated levels. Such a system can also pose several challenges, including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation. We subscribe to the view that no model can faithfully represent reality, but detailed, descriptive models are useful in learning about the system and bringing about a qualitative jump in understanding of the system it attempts to model – provided they are properly validated. Our own approach to model evaluation is to consider the entire life cycle and assess the validity under two broad dimensions of (1) internally focused validity/quality achieved through structural, methodological, and ontological evaluations; and (2) external validity consisting of micro validity, macro validity, and qualitative, causal and narrative validity. In this paper, we also elaborate on selected validation techniques that we have employed in the past. We recommend a triangulation of multiple validation techniques, including methodological soundness, qualitative validation techniques, such as face validation by experts and narrative validation, and formal validation tests, including correspondence testing.

Self-exciting point process models for political conflict forecasting

“…Innovative Decisions, Inc. produced a Naive Bayesian network structure to combine a set of social science models [14]. Lockheed Martin also developed conflict forecasting models using ICEWS data; the company implemented six models including logistic regression models, Bayesian methods, geospatial network models, and agent-based models [3,6,19].…”

A stochastic agent-based model of pathogen propagation in dynamic multi-relational social networks

“…Having defined the distance between corresponding distributions in the two models, we use the L ∞ norm to extend to a de nition of distance between statistical network models : $${\Delta }^{\ast }({\mathcal{M}}_1,{\mathcal{M}}_2)\stackrel{\mathrm{def}}{=}\max true(\underset{i=1}{\overset{m}{\max }}\Delta ({\alpha }_i,{\alpha }_i^{\prime }),\underset{j=1}{\overset{l}{\max }}\underset{i=1}{\overset{m}{\max }}\Delta ({\beta }_{i,j},{\beta }_{i,j}^{\prime }),\underset{j=1}{\overset{l}{\max }}\Delta ({\mathbb{X}}_j,{\mathbb{X}}_j^{\prime })\underset{j=1}{\overset{l}{\max }}\Delta (\stackrel{=}{{\chi }_j},\stackrel{=}{{\chi }_j^{\prime }},true)$$ By considering the worst-case divergences of all constituent distributions within the two models, we hope to produce a holistic assessment of the relative validity of each model against the other [Bharathy and Silverman, 2013]. …”

“…By considering the worst-case divergences of all constituent distributions within the two models, we hope to produce a holistic assessment of the relative validity of each model against the other. 60…”

User Behavior Model Based on Affordances and Emotions: A New Approach for an Optimal Use Method in Product–User Interactions

“…However, the complex settings for too many variables entail difficulties in understanding the interaction between users and surroundings by steps. In addition, it is not clear how such variables can reflect real situations, and its validation issues are sometimes raised (Bharathy & Silverman, 2013). Though the PMFserv model has been utilized in many simulation fields such as the military, robots, and evacuations (Meador & Hill, 2011;Nye & Silverman, 2013;Silverman et al, 2012), it has not been properly applied to the analyses of the interaction between users and products in daily lives and the human behaviors in stages.…”