A taxonomy for reproducible and replicable research in environmental modelling

Essawy, Bakinam T.; Goodall, Jonathan L.; Voce, Daniel; Morsy, Mohamed M.; Sadler, Jeffrey M.; Choi, Young Don; Tarboton, David G.; Malik, Tanu

doi:10.1016/j.envsoft.2020.104753

Cited by 27 publications

(15 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(4) It enables reproducible-and therefore more credible-computational research by fully documenting simulation experiments, including the exact input, code, and quantitative analyses, and all technical details of the experiments, which are likely to be lost if not recorded promptly. While reproducibility in complex simulation studies can be achieved by tools that encapsulate the end-to-end workflow, from raw data to final publication-ready outputs (e.g., containerized virtual environments), documentation that fully describes the analysis is fundamental (Essawy et al, 2020). (5) Importantly, it allows modellers to easily assemble and produce TRACE documentation of their model (i.e., "writing your TRACE document in 15 min per day").…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

Ayllón¹,

Railsback²,

Gallagher³

et al. 2021

Environmental Modelling & Software

View full text Add to dashboard Cite

The acceptance and usefulness of simulation models are often limited by the efficiency, transparency, reproducibility, and reliability of the modelling process. We address these issues by suggesting that modellers (1) "trace" the iterative modelling process by keeping a modelling notebook corresponding to the laboratory notebooks used by empirical researchers, (2) use a standardized notebook structure and terminology based on the existing TRACE documentation framework, and (3) use their notebooks to compile TRACE documents that supplement publications and reports. These practices have benefits for model developers, users, and stakeholders: improved and efficient model design, analysis, testing, and application; increased model acceptance and reuse; and replicability and reproducibility of the model and the simulation experiments. Using TRACE terminology and structure in modelling notebooks facilitates production of TRACE documents. We explain the

show abstract

Section: Discussionmentioning

confidence: 99%

“…Design a reproducible research workflow. When performing large or repeated simulation experiments, develop, document, and automate end-to-end workflows from raw inputs to publication-ready outputs (Kitzes et al, 2018;Essawy et al, 2020).…”

Section: Recommendationsmentioning

confidence: 99%

Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

Ayllón¹,

Railsback²,

Gallagher³

et al. 2021

Environmental Modelling & Software

View full text Add to dashboard Cite

show abstract

“…Because replicating studies with new independent data is expensive, rarely published in high-impact journals, and sometimes even methodologically impossible, computationally reproducible research (most often termed simply "reproducible research") is often suggested as a pathway for increasing our ability to assess the validity and rigor of scientific results (Peng 2011). Research is reproducible when others can reproduce the results of a scientific study given only the original data, code, and documentation (Essawy et al 2020). This approach focuses on the research process after data collection is complete, and it has many (though not all) of the advantages of replicating studies with independent data while minimizing the largest barrier (i.e., the financial and time costs of collecting new data).…”

Section: Introductionmentioning

confidence: 99%

A Beginner's Guide to Conducting Reproducible Research

Alston

Rick

2021

Bulletin Ecologic Soc America

View full text Add to dashboard Cite

“…2018; Essawy et al. 2020). It is suspected that reproducibility and scientific provenance (i.e., a digital record of what data and methods were used to produce the results and conclusions) will be a key review criterion for future geoscience publications (Gil et al.…”

Section: Introductionmentioning

confidence: 99%

Visualization Workflow for Quantifying Parameter Sensitivities and Uncertainties for Hydrologic Models

Finkenbiner

Semmendinger

2021

J American Water Resour Assoc

View full text Add to dashboard Cite

From regional to continental scales, hydrologic processes are represented by modular modeling frameworks dependent on input datasets and parameter sets representing physical attributes. The hydrologic community needs a common procedure to evaluate model output based on parameter sensitivities and uncertainties compared to performance metrics (i.e., objective functions), especially for large parameter sets. We developed a reproducible workflow for evaluating hydrologic models to objectively analyze model outputs as a function of parameter choice using numerical and visualization techniques. Our workflow was implemented on three separate case studies, each with a different hydrologic model, and the results can be reproduced and visualized from a community github code repository. Model parameter sensitivity was evaluated using several global sensitivity indices and Bayesian theory. Uncertainty in parameter spaces was quantified to highlight the impact of unreliable input data on model output. Model parameter sensitivities and uncertainties were evaluated numerically and visually to provide a comprehensive perspective on their impacts on model output. For each case study, we provided a summary and interpretation of the workflow results. Our workflow can be integrated into hydrologic modeling frameworks for objective modular model and parameter set evaluations based on a data‐driven approach appropriate for model selection.

show abstract

A taxonomy for reproducible and replicable research in environmental modelling

Cited by 27 publications

References 34 publications

Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

A Beginner's Guide to Conducting Reproducible Research

Visualization Workflow for Quantifying Parameter Sensitivities and Uncertainties for Hydrologic Models

Contact Info

Product

Resources

About