Does expertise matter in replication? An examination of the reproducibility project: Psychology

Bench, Shane W.; Rivera, Grace N.; Schlegel, Rebecca J.; Hicks, Joshua A.; Lench, Heather C.

doi:10.1016/j.jesp.2016.07.003

“…This is consistent with our expectations that all research is not created equally and that replication studies, especially by novice investigators, may not be executed with same meticulousness as the original research. Our analyses also are consistent with a recent study that found that replications run by high-expertise teams, with 10 or more publications, produced effect sizes that were nearly twice as large as those obtained by low-expertise teams, in part due to wiser choices of the specific method to replicate (Bench et al, 2017). …”

Section: The Impact Of New Investigatorssupporting

confidence: 90%

How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions

Cunningham

¹

,

Baumeister

²

2016

View full text Add to dashboard Cite

The limited resource model states that self-control is governed by a relatively finite set of inner resources on which people draw when exerting willpower. Once self-control resources have been used up or depleted, they are less available for other self-control tasks, leading to a decrement in subsequent self-control success. The depletion effect has been studied for over 20 years, tested or extended in more than 600 studies, and supported in an independent meta-analysis (Hagger et al., 2010). Meta-analyses are supposed to reduce bias in literature reviews. Carter et al.'s (2015) meta-analysis, by contrast, included a series of questionable decisions involving sampling, methods, and data analysis. We provide quantitative analyses of key sampling issues: exclusion of many of the best depletion studies based on idiosyncratic criteria and the emphasis on mini meta-analyses with low statistical power as opposed to the overall depletion effect. We discuss two key methodological issues: failure to code for research quality, and the quantitative impact of weak studies by novice researchers. We discuss two key data analysis issues: questionable interpretation of the results of trim and fill and Funnel Plot Asymmetry test procedures, and the use and misinterpretation of the untested Precision Effect Test and Precision Effect Estimate with Standard Error (PEESE) procedures. Despite these serious problems, the Carter et al. (2015) meta-analysis results actually indicate that there is a real depletion effect -contrary to their title.

show abstract

“…35 28 3.6. 5 Supporting information 46 38 6 Acknowledgments 58 39 − log A p(y) = log A 1 p(y) (2) which quantifies the number of questions with A possible answers that we would need to ask to 97 determine y. The logarithm's base, A, could have any value, but we will typically assume that A = 2 98 and therefore that information is measured in "bits", i.e.…”

mentioning

confidence: 99%

A mathematical theory of knowledge, science, bias and pseudoscience

Fanelli

¹

2016

Preprint

0

View full text Add to dashboard Cite

This article proposes quantitative answers to meta-scientific questions including "how much knowledge is attained by a research field?", "how rapidly is a field making progress?", "what is the expected reproducibility of a result?", "how much knowledge is lost from scientific bias and misconduct?" "what do we mean by soft science?", "what demarcates a pseudoscience?".Knowledge is suggested to be a system-specific property measured by K, a quantity determined by how much of the information contained in an explanandum is compressed by an explanans, which is composed of an information "input" and a "theory/methodology" conditioning factor. This approach is justified on three grounds: 1) K is derived from postulating that information is finite and knowledge is information compression; 2) K is compatible and convertible to ordinary measures of effect size and algorithmic complexity; 3) K is physically interpretable as a measure of entropic efficiency. Moreover, the K function has useful properties that support its potential as a measure of knowledge. Examples given to illustrate the possible uses of K include: the knowledge value of proving Fermat's last theorem; the accuracy of measurements of the mass of the electron; the half life of predictions of solar eclipses; the usefulness of evolutionary models of reproductive skew; the significance of gender differences in personality; the sources of irreproducibility in psychology; the impact of scientific misconduct and questionable research practices; the knowledge value of astrology. Furthermore, measures derived from K may complement ordinary meta-analysis and may give rise to a universal classification of sciences and pseudosciences.Simple and memorable mathematical formulae that summarize the theory's key results may find practical uses in meta-research, philosophy and research policy.PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.1968v5 | CC BY 4.0 Open Access | rec:A science of science is flourishing in all disciplines and promises to boost discovery on all research 41 fronts [1]. Commonly branded "meta-science" or "meta-research", this rapidly expanding literature 42 of empirical studies, experiments, interventions, and theoretical models explicitly aims to take a 43 "bird's eye view" of science and a decidedly cross-disciplinary approach to studying the scientific 44 method, which is dissected and experimented upon as any other topic of academic inquiry. To fully 45 mature into an independent field, meta-research needs a fully cross-disciplinary, quantitative, and 46 operationalizable theory of scientific knowledge -a unifying paradigm that, in simple words, can help 47 tell apart "good" from "bad" science. 48This article proposes such a meta-scientific theory and methodology. By means of analyses and 49 practical examples, it suggests that a system-specific quantity named "K" can help answer 50 meta-scientific questions including "how much knowledge is attained by a research field?", "how 51 rapidly is a field making progress?", "what is the expected ...

show abstract

“…35 28 3.6. 5 Supporting information 46 38 6 Acknowledgments 58 39 − log A p(y) = log A 1 p(y) (2) which quantifies the number of questions with A possible answers that we would need to ask to 97 determine y. The logarithm's base, A, could have any value, but we will typically assume that A = 2 98 and therefore that information is measured in "bits", i.e.…”

mentioning

confidence: 99%

“…n Y 39 log 10 ≈ 0.957 (38) with the last approximation due to the case that the value can be stored and used for a very 643 large n Y times, yielding h ≈ 1. More accurate calculations would require estimating the h 644 component, too.…”

mentioning

confidence: 99%

“…This conclusion, however, was 991 questioned on various grounds, including: limitations in current statistical approaches used to 992 predict and estimate reproducibility (e.g. [34][35][36]), methodological differences between original and 993 replication studies [37], variable expertise of the replicators [38], and variable contextual sensitivity 994 of the phenomena studied [39,40]. The common element behind all these concerns is that the 995 replication study was not actually identical to the original, but diverged in details that affected the 996 results uni-directionally.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Untitled

Supplemental Information 1: R Code

0

View full text Add to dashboard Cite

This article proposes quantitative answers to meta-scientific questions including "how much knowledge is attained by a research field?", "how rapidly is a field making progress?", "what is the expected reproducibility of a result?", "how much knowledge is lost from scientific bias and misconduct?" "what do we mean by soft science?", "what demarcates a pseudoscience?".Knowledge is suggested to be a system-specific property measured by K, a quantity determined by how much of the information contained in an explanandum is compressed by an explanans, which is composed of an information "input" and a "theory/methodology" conditioning factor. This approach is justified on three grounds: 1) K is derived from postulating that information is finite and knowledge is information compression; 2) K is compatible and convertible to ordinary measures of effect size and algorithmic complexity; 3) K is physically interpretable as a measure of entropic efficiency. Moreover, the K function has useful properties that support its potential as a measure of knowledge. Examples given to illustrate the possible uses of K include: the knowledge value of proving Fermat's last theorem; the accuracy of measurements of the mass of the electron; the half life of predictions of solar eclipses; the usefulness of evolutionary models of reproductive skew; the significance of gender differences in personality; the sources of irreproducibility in psychology; the impact of scientific misconduct and questionable research practices; the knowledge value of astrology. Furthermore, measures derived from K may complement ordinary meta-analysis and may give rise to a universal classification of sciences and pseudosciences.Simple and memorable mathematical formulae that summarize the theory's key results may find practical uses in meta-research, philosophy and research policy.PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.1968v5 | CC BY 4.0 Open Access | rec:A science of science is flourishing in all disciplines and promises to boost discovery on all research 41 fronts [1]. Commonly branded "meta-science" or "meta-research", this rapidly expanding literature 42 of empirical studies, experiments, interventions, and theoretical models explicitly aims to take a 43 "bird's eye view" of science and a decidedly cross-disciplinary approach to studying the scientific 44 method, which is dissected and experimented upon as any other topic of academic inquiry. To fully 45 mature into an independent field, meta-research needs a fully cross-disciplinary, quantitative, and 46 operationalizable theory of scientific knowledge -a unifying paradigm that, in simple words, can help 47 tell apart "good" from "bad" science. 48This article proposes such a meta-scientific theory and methodology. By means of analyses and 49 practical examples, it suggests that a system-specific quantity named "K" can help answer 50 meta-scientific questions including "how much knowledge is attained by a research field?", "how 51 rapidly is a field making progress?", "what is the expected ...

show abstract

Does expertise matter in replication? An examination of the reproducibility project: Psychology

Cited by 23 publications

References 8 publications

How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions

How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions

A mathematical theory of knowledge, science, bias and pseudoscience

Untitled

Contact Info

Product

Resources

About