Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

Abstract

The acceptance and usefulness of simulation models are often limited by the efficiency, transparency, reproducibility, and reliability of the modelling process. We address these issues by suggesting that modellers (1) "trace" the iterative modelling process by keeping a modelling notebook corresponding to the laboratory notebooks used by empirical researchers, (2) use a standardized notebook structure and terminology based on the existing TRACE documentation framework, and (3) use their notebooks to compile TRACE documents that supplement publications and reports. These practices have benefits for model developers, users, and stakeholders: improved and efficient model design, analysis, testing, and application; increased model acceptance and reuse; and replicability and reproducibility of the model and the simulation experiments. Using TRACE terminology and structure in modelling notebooks facilitates production of TRACE documents. We explain the

Simulation Studies of Social Systems – Telling the Story Based on Provenance Patterns

“…Starting from this question, a psychological experiment was designed (Design Data Collection, ddc2), using (ML2; referring to Briñol and Petty [46]) and ML3 (referring to Wintle et al [47]) as methodology literature (ML) inputsone satisfying the ML input of the pattern, and the other serving as an optional additional input (X). The results of this activity are a data collection procedure (CP2; referring to the survey 6 ), the preregistration (PR2; linking to the preregistration stored on OSF 7 ) and the ethical approval (E2; referring to the University of Southampton Ethics Committee, ERGO number 56865). Similarly, cpd2 and ad2 match the patterns Collecting Primary Data and Analyzing Data.…”

“…Their documentation therefore requires significant effort and has been subject of reporting guidelines such as [1]- [3]. Computational support for recording crucial information, such as data, simulation models, assumptions, research questions etc., about simulation studies includes adopting archives [4], Wikis [5], electronic notebooks [6], as well as provenance standards (and a graph-based database) [7].…”

High-Fidelity Agent-Based Modeling to Support Prevention Decision-Making: an Open Science Approach

“…The recently developed TRAnsparent and Comprehensive Ecological modeling documentation guidelines, or in short TRACE (Grimm et al, 2014 ), support the reporting of the model-building processes by presenting model builders a structure and language to adopt in their documentation of activities undertaken as part of the building process. TRACE asks modelers to keep a notebook of their model-building efforts and organize it using 8 main elements: Problem formulation, Model description, Data evaluation, Conceptual model evaluation, Implementation verification, Model output verification, Model analysis and application, and Model output corroboration (Ayllón et al, 2021 ; Grimm et al, 2014 ). This structure can then be leveraged to present an overview of efforts.…”

Toward a complete interdisciplinary treatment of scale

“…Software practices include the Architectural Decision Records (Emery and Hilliard, 2008;Zdun et al, 2014), which advocate storing such documentation alongside code in version control. Likewise, the TRACE documentation framework suggests keeping "computational notebooks" in version control as a complement to traditional "pen-and-paper" notebooks with similar aims of documenting decisions made throughout the modeling (Ayllón et al, 2021).…”

“…Specific to scale choices, the level of shared understanding and other major considerations could be explicitly catalogued in a "core" table. This table would detail the spatial and temporal scales (Koo et al, 2020), knowledge sources (Kragt et al, 2013), expected computational requirements, major uncertainty sources (Refsgaard et al, 2007;van der Sluijs, 2007;Reichert, 2020), the relevant system(s) affected, and the modeling process (Hutton et al, 2016;Ayllón et al, 2021). The ODD protocol (Grimm et al, 2006(Grimm et al, , 2010 was used to capture these considerations in the GPSCA study, adoption of which mandates that pertinent aspects of scale and their representations are documented.…”