2022
DOI: 10.1111/cgf.14443
|View full text |Cite
|
Sign up to set email alerts
|

A Survey of Tasks and Visualizations in Multiverse Analysis Reports

Abstract: Analysing data from experiments is a complex, multi‐step process, often with multiple defensible choices available at each step. While analysts often report a single analysis without documenting how it was chosen, this can cause serious transparency and methodological issues. To make the sensitivity of analysis results to analytical choices transparent, some statisticians and methodologists advocate the use of ‘multiverse analysis’: reporting the full range of outcomes that result from all combinations of defe… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
1
1
1

Citation Types

1
33
0

Year Published

2022
2022
2024
2024

Publication Types

Select...
5
4

Relationship

0
9

Authors

Journals

citations
Cited by 19 publications
(34 citation statements)
references
References 84 publications
1
33
0
Order By: Relevance
“…The converging findings of these studies across modalities suggest that it is important to test the robustness of reported results to specific analytical choices. One proposed solution to tackle the analytical variability, where many different analytical approaches are compared, is multiverse analysis (Hall et al 2022). There are two broad types of multiverse tools.…”
Section: Multiverse Analysismentioning
confidence: 99%
“…The converging findings of these studies across modalities suggest that it is important to test the robustness of reported results to specific analytical choices. One proposed solution to tackle the analytical variability, where many different analytical approaches are compared, is multiverse analysis (Hall et al 2022). There are two broad types of multiverse tools.…”
Section: Multiverse Analysismentioning
confidence: 99%
“…One solution that has been suggested is "multiverse analysis" (99,100), in which a range of theoretically and statistically reasonable analysis pipelines are tested and reported. This concept has been used across fields, sometimes under different names (e.g., specification curve analysis (101,102)).…”
Section: Robustness To Analytical Variability: Same Data Different Me...mentioning
confidence: 99%
“…Third, multiverse analysis is complex, making it challenging to integrate, visualize, and summarize findings and conclusions. Diverse approaches to visualization and reporting have been developed (99,101,107,108). One statistical approach for interpretation of multiverse analysis in fMRI is a "consensus analysis", which is a type of meta-analysis over the statistical maps resulting from the different pipelines, adapted to account for their dependency (since they are based on the same data) (86,109).…”
mentioning
confidence: 99%
“…A full multiverse analysis should then be followed by a full report of the results from all possible combinations of forking paths. This report might be, for example a p -value distribution, a specification curve analysis and plot, or a vibration of effects plot (for review, see Hall et al, 2022). A joint conclusion can then be drawn by integrating the results across the multiverse of options (Simonsohn et al, 2020) or by reporting a multiverse variability index that quantifies the robustness of the results across the forking paths (Olsson-collentine et al, 2023).…”
Section: Introductionmentioning
confidence: 99%