Learning from reproducing computational results: introducing three principles and the<i>Reproduction Package</i>

Krafczyk, Matthew; Shi, August; Bhaskar, Adhithya; Marinov, Darko; Stodden, Victoria

doi:10.1098/rsta.2020.0069

Cited by 19 publications

(13 citation statements)

References 103 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[ 107,108 ] Our criteria thus meant that we could only assess a subset of existing models and it is possible that different trends or initial error levels are observed in other models. We note that projects that shared their code were also transparent regarding how their computational results were produced, [ 109 ] hence we were able to perform verification and we applied the same level of transparency when conveying the model's parameters.…”

Section: Discussionmentioning

confidence: 99%

When Do We Need Massive Computations to Perform Detailed COVID‐19 Simulations?

Lutz

Giabbanelli

2021

Advcd Theory and Sims

View full text Add to dashboard Cite

The COVID‐19 pandemic has infected over 250 million people worldwide and killed more than 5 million as of November 2021. Many intervention strategies are utilized (e.g., masks, social distancing, vaccinations), but officials making decisions have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also require significant processing power or time. It is examined whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories resembling the original simulation results. Using four previously published agent‐based models (ABMs) for COVID‐19, a decision tree regression for each ABM is built and its predictions are compared to the corresponding ABM. Accurate machine learning meta‐models are generated from ABMs without strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data: the root‐mean‐square error (RMSE) with 25% of the data is close to the RMSE for the full dataset (0.15 vs 0.14 in one model; 0.07 vs 0.06 in another). However, meta‐models for ABMs employing strong interventions require much more training data (at least 60%) to achieve a similar accuracy. In conclusion, machine learning meta‐models can be used in some scenarios to assist in faster decision‐making.

show abstract

Section: Discussionmentioning

confidence: 99%

When Do We Need Massive Computations to Perform Detailed COVID‐19 Simulations?

Lutz

Giabbanelli

2021

Advcd Theory and Sims

View full text Add to dashboard Cite

show abstract

“…So, how do we ensure that we publish scholarship, not just advertising? Acknowledging that more detailed guidelines exist in the literature on computational reproducibility [18], we propose to begin with the following simple principles:…”

Section: Daring: Challenging Current Conceptionsmentioning

confidence: 99%

Sharing and Caring: Creating a Culture of Constructive Criticism in Computational Legal Studies

Coupette¹,

Hartung²

2022

Preprint

View full text Add to dashboard Cite

We introduce seven foundational principles for creating a culture of constructive criticism in computational legal studies. Beginning by challenging the current perception of papers as the primary scholarly output, we call for a more comprehensive interpretation of publications. We then suggest to make these publications computationally reproducible, releasing all of the data and all of the code all of the time, on time, and in the most functioning form possible. Subsequently, we invite constructive criticism in all phases of the publication life cycle. We posit that our proposals will help form our field, and float the idea of marking this maturity by the creation of a modern flagship publication outlet for computational legal studies.

show abstract

“…The ongoing reproducibility crisis, undermining the foundation of science 1 , raises various concerns ranging from study design to statistical rigor 2 , 3 . Population genetics is confounded by its utilization of small sample sizes, ignorance of effect sizes, and adoption of questionable study designs.…”

Section: Introductionmentioning

confidence: 99%

Principal Component Analyses (PCA)-based findings in population genetic studies are highly biased and must be reevaluated

Elhaik

2022

Sci Rep

103

View full text Add to dashboard Cite

Principal Component Analysis (PCA) is a multivariate analysis that reduces the complexity of datasets while preserving data covariance. The outcome can be visualized on colorful scatterplots, ideally with only a minimal loss of information. PCA applications, implemented in well-cited packages like EIGENSOFT and PLINK, are extensively used as the foremost analyses in population genetics and related fields (e.g., animal and plant or medical genetics). PCA outcomes are used to shape study design, identify, and characterize individuals and populations, and draw historical and ethnobiological conclusions on origins, evolution, dispersion, and relatedness. The replicability crisis in science has prompted us to evaluate whether PCA results are reliable, robust, and replicable. We analyzed twelve common test cases using an intuitive color-based model alongside human population data. We demonstrate that PCA results can be artifacts of the data and can be easily manipulated to generate desired outcomes. PCA adjustment also yielded unfavorable outcomes in association studies. PCA results may not be reliable, robust, or replicable as the field assumes. Our findings raise concerns about the validity of results reported in the population genetics literature and related fields that place a disproportionate reliance upon PCA outcomes and the insights derived from them. We conclude that PCA may have a biasing role in genetic investigations and that 32,000-216,000 genetic studies should be reevaluated. An alternative mixed-admixture population genetic model is discussed.

show abstract

Learning from reproducing computational results: introducing three principles and theReproduction Package

Cited by 19 publications

References 103 publications

When Do We Need Massive Computations to Perform Detailed COVID‐19 Simulations?

When Do We Need Massive Computations to Perform Detailed COVID‐19 Simulations?

Sharing and Caring: Creating a Culture of Constructive Criticism in Computational Legal Studies

Principal Component Analyses (PCA)-based findings in population genetic studies are highly biased and must be reevaluated

Contact Info

Product

Resources

About