Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)

Heathers, James; Anaya, Jordan; Zee, Tim van der; Brown, Nicholas J. L.

doi:10.7287/peerj.preprints.26968v1

Cited by 33 publications

(33 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A. Heathers, Anaya, Zee, & Brown, 2018), GRIM (N. J. L. Brown & Heathers, 2016), and GRIMMER (Anaya, 2016) are some examples of other statistical methods that test for problematic or fabricated summary statistics (see also Buyse et al, 1999). However, these methods were not applicable in the studies we presented, because they require ordinal scale measures.…”

Section: Discussionmentioning

confidence: 99%

Detection of data fabrication using statistical tools

Hartgerink¹,

Voelkel²,

Wicherts³

et al. 2019

Preprint

View full text Add to dashboard Cite

Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Study 1, we tested the validity of statistical methods to detect fabricated data at the study level using summary statistics. Using (arguably) genuine data from the Many Labs 1 project on the anchoring effect (k=36) and fabricated data for the same effect by our participants (k=39), we tested the validity of our newly proposed 'reversed Fisher method', variance analyses, and extreme effect sizes, and a combination of these three indicators using the original Fisher method. Results indicate that the variance analyses perform fairly well when the homogeneity of population variances is accounted for and that extreme effect sizes perform similarly well in distinguishing genuine from fabricated data. The performance of the 'reversed Fisher method' was poor and depended on the types of tests included. In Study 2, we tested the validity of statistical methods to detect fabricated data using raw data. Using (arguably) genuine data from the Many Labs 3 project on the classic Stroop task (k=21) and fabricated data for the same effect by our participants (k=28), we investigated the performance of digit analyses, variance analyses, multivariate associations, and extreme effect sizes, and a combination of these four methods using the original Fisher method. Results indicate that variance analyses, extreme effect sizes, and multivariate associations perform fairly well to excellent in detecting fabricated data using raw data, while digit analyses perform at chance levels. The two studies provide mixed results on how the use of random number generators affects the detection of data fabrication. Ultimately, we consider the variance analyses, effect sizes, and multivariate associations valuable tools to detect potential data anomalies in empirical (summary or raw) data. However, we argue against widespread (possible automatic) application of these tools, because some fabricated data may be irregular in one aspect but not in another. Considering how violations of the assumptions of fabrication detection methods may yield high false positive or false negative probabilities, we recommend comparing potentially fabricated data to genuine data on the same topic.

show abstract

Section: Discussionmentioning

confidence: 99%

Detection of data fabrication using statistical tools

Hartgerink¹,

Voelkel²,

Wicherts³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…As the success of StatCheck demonstrates, even relatively simple automated validation methods that can be implemented given present standards and technologies can have a large impact on the quality of social science research. Accordingly, one natural target for near-term investment in this area is development of centralized, automated versions of existing quality assessment tools-e.g., various QRP and bias detection methods (Brown & Heathers, 2016;e.g., Gerber & Malhotra, 2008;Heathers, Anaya, van der Zee, & Brown, 2018;Simonsohn, Nelson, & Simmons, 2014), reporting checklists, etc.-that are presently deployed manually and sporadically. More sophisticated kinds of validation-e.g., the ability to automatically identify errors in formal mathematical models-will likely improve progressively in the coming decades as computer science, statistics and social science make further advances.…”

Section: Discussionmentioning

confidence: 99%

Enhancing and accelerating social science via automation: Challenges and opportunities

Yarkoni¹,

Eckles²,

Heathers³

et al. 2019

Preprint

View full text Add to dashboard Cite

Automation plays an increasingly important role in science, but the social sciences have been comparatively slow to take advantage of emerging technologies and methods. In this review, we argue that greater investment in automation would be one of the most effective and cost-effective ways to boost the reliability, validity, and utility of social science research. We identify five core areas ripe for potentially transformative investment, including (1) machine-readable standards, (2) data access platforms, (3) search and discoverability, (4) claim validation, and (5) insight generation. In each case, we review limitations associated with current practices, identify concrete opportunities for improvement via automation, and discuss near-term barriers to progress. We conclude with a discussion of practical and ethical considerations researchers will need to keep in mind when working to enhance and accelerate social science progress via automation.

show abstract

“…Despite testing various statistical methods to detect data fabrication, we did not test all available statistical methods to detect data fabrication in summary statistics. SPRITE (Heathers et al 2018), GRIM (Brown and Heathers 2016), and GRIMMER (Anaya 2016) are some examples of other statistical methods that test for problematic or fabricated summary statistics (see also Buyse et al 1999). However, these methods were not applicable in the studies we presented, because they require ordinal scale measures.…”

Section: Discussionmentioning

confidence: 99%

Contributions towards understanding and building sustainable science

Hartgerink¹

2019

Preprint

View full text Add to dashboard Cite

This dissertation focuses on either understanding and detecting threats to the epistemology of science (chapters 1-6) or making practical advances to remedy epistemological threats (chapters 7-9). A digital version of this dissertation is available at https://phd.chjh.nlChapter 1 reviews the literature on responsible conduct of research, questionable research practices, and research misconduct. Chapter 2 reanalyzes Head et al (2015) their claims about widespread p-hacking for robustness. Chapter 3 examines 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of $p$-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. Chapter 4 examines evidence for false negatives in nonsignificant results throughout psychology, gender effects, and the Reproducibility Project: Psychology. Chapter 5 describes a dataset that is the result of content mining 167,318 published articles for statistical test results reported according to the standards prescribed by the American Psychological Association (APA). In Chapter 6, I test the validity of statistical methods to detect fabricated data in two studies. Chapter 7 tackles the issue of data extraction from figures in scholarly publications. In Chapter 8 I argue that "after-the-fact" research papers do not help alleviate issues of access, selective publication, and reproducibility, but actually cause some of these threats because the chronology of the research cycle is lost in a research paper. I propose to give up the academic paper and propose a digitally native "as-you-go" alternative. In Chapter 9 I propose a technical design for this.

show abstract

Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)

Cited by 33 publications

References 24 publications

Detection of data fabrication using statistical tools

Detection of data fabrication using statistical tools

Enhancing and accelerating social science via automation: Challenges and opportunities

Contributions towards understanding and building sustainable science

Contact Info

Product

Resources

About