Looking Our Limitations in the Eye: A Call for More Thorough and Honest Reporting of Study Limitations

Clarke, Beth; Alley, Lindsay; Ghai, Sakshi; Flake, Jessica Kay; Rohrer, Julia M.; Simmons, Joseph P.; Schiavone, Sarah R.; Vazire, Simine

doi:10.31234/osf.io/386bh

Cited by 3 publications

(1 citation statement)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Outcome Set [31][32][33] Alternatively, the omission of these details and considerations could be explicitly discussed as a study limitation (e.g. see recommendations for discussing limitations specific to outcome and outcome measure choices 35 ).…”

Section: Discussionmentioning

confidence: 99%

Emotion constructs and outcome measures following false positive breast screening test results: A systematic review of reporting clarity and selection rationale

Long,

Hindmarch,

Martindale

et al. 2024

Psycho-Oncology

View full text Add to dashboard Cite

Objective(i) To systematically identify constructs and outcome measures used to assess the emotional and mood impact of false positive breast screening test results; (ii) to appraise the reporting clarity and rationale for selecting constructs and outcome measures.MethodsDatabases (MEDLINE, CINAHL, PsycINFO) were systematically searched from 1970. Studies using standardised and non‐standardised outcome measures to evaluate the emotion or mood impact of false positive breast screening test results were eligible. A 15‐item coding scheme was devised to appraise articles on clarity and rationale for selected constructs and measures.ResultsForty‐seven articles were identified. The most investigated constructs were general anxiety and depression and disease‐specific anxiety and worry. Twenty‐two standardised general outcome questionnaire measures and three standardised disease‐specific outcome questionnaire measures were identified. Twenty articles used non‐standardised scales/items. Reporting of constructs and outcome measures was generally clear, but rationales for their selection were lacking. Anxiety was typically justified, but justification for depression was almost always absent. Practical and psychometric justification for selecting outcome measures was lacking, and theoretical rationale was absent.ConclusionsHeterogeneity in constructs and measures, coupled with unclear rationale for these, impedes a thorough understanding of why there are emotional effects of false positive screening test results. This may explain the repeated practice of investigating less relevant outcomes such as depression. There is need to develop a consensual conceptual model of and standardised approach to measuring emotional impact from cancer screening test results, to address heterogeneity and other known issues of interpreting an inconsistent evidence base.

show abstract

Section: Discussionmentioning

confidence: 99%

Emotion constructs and outcome measures following false positive breast screening test results: A systematic review of reporting clarity and selection rationale

Long,

Hindmarch,

Martindale

et al. 2024

Psycho-Oncology

View full text Add to dashboard Cite

show abstract

Assessing transparency and methodological precision in variable measurement within organizational research: implications for validity

Øby

2024

Qual Quant

View full text Add to dashboard Cite

This study addresses the call for enhanced transparency in methodological reporting by critically assessing methodological rigor and transparency, particularly in variable measurement, within primary cross-sectional survey-based research in organizational studies. Through a combined approach of scoping review for systematic literature identification and qualitative analysis, instances of inadequate reporting transparency and lack of methodological rigor are revealed. Analyzing data from 68 published studies, the findings highlight unclear construct definitions and a consistent absence of theoretical limitations when presenting study findings, pointing to a need for better reporting transparency. Additionally, the study uncovers the presence of elusive links between variable constructs and measures, and misplaced claims of prior validation for measurement instruments, indicating a lack of methodological rigor. Collectively, these findings challenge the interpretation and validity of research results, potentially leading to misconceptions and misinterpretations about what is known about the variable(s) in question. In turn, this may lead to researchers and practitioners basing their research or practice on flawed assumptions about what a group of studies is saying or indicating. While acknowledging prior recommendations on variable measurement and transparency, this study brings renewed focus to these areas. It also explores the role of measurement proficiency, debating whether the observed limitations reflect ethical concerns or are indicative of researchers’ capabilities in variable measurement, validity assessment, and reporting.

show abstract

Causal inference for psychologists who think that causal inference is not for them

Rohrer

2024

Social & Personality Psych

Self Cite

View full text Add to dashboard Cite

Correlation does not imply causation and psychologists' causal inference training often focuses on the conclusion that therefore experiments are needed—without much consideration for the causal inference frameworks used elsewhere. This leaves researchers ill‐equipped to solve inferential problems that they encounter in their work, leading to mistaken conclusions and incoherent statistical analyses. For a more systematic approach to causal inference, this article provides brief introductions to the potential outcomes framework—the “lingua franca” of causal inference—and to directed acyclic graphs, a graphical notation that makes it easier to systematically reason about complex causal situations. I then discuss two issues that may be of interest to researchers in social and personality psychology who think that formalized causal inference is of little relevance to their work. First, posttreatment bias: In various common scenarios (noncompliance, mediation analysis, missing data), researchers may analyze data from experimental studies in a manner that results in internally invalid conclusions, despite randomization. Second, tests of incremental validity: Routine practices in personality psychology suggest that they may be conducted for at least two different reasons (to demonstrate the non‐redundancy of new scales, to support causal conclusions) without being particularly suited for either purpose. Taking causal inference seriously is challenging; it reveals assumptions that may make many uncomfortable. However, ultimately it is a necessary step to ensure the validity of psychological research.

show abstract

Looking Our Limitations in the Eye: A Call for More Thorough and Honest Reporting of Study Limitations

Cited by 3 publications

References 70 publications

Emotion constructs and outcome measures following false positive breast screening test results: A systematic review of reporting clarity and selection rationale

Emotion constructs and outcome measures following false positive breast screening test results: A systematic review of reporting clarity and selection rationale

Assessing transparency and methodological precision in variable measurement within organizational research: implications for validity

Causal inference for psychologists who think that causal inference is not for them

Contact Info

Product

Resources

About